Techniques for neuromuscular-signal-based detection of in-air hand gestures for text production and modification, and systems, wearable devices, and methods for using these techniques

ABSTRACT

The various implementations described herein include methods and systems for producing and modifying text using neuromuscular-signal-sensing devices. In one aspect, a method includes causing the display of a plurality of text terms input by a user. Using data from one or more neuromuscular-signal sensors in communication with the wearable device, an in-air hand gesture performed by the user is detected while the text terms are displayed. In response to the in-air hand gesture, a text-modification mode is enabled that allows for modifying the text terms input by the user. A target term is identified and, while the text-modification mode is enabled, data about a voice input provided by the user for modifying the target term is received. The method further includes causing a modification to the target term in accordance with the voice input from the user.

PRIORITY AND RELATED APPLICATIONS

This application claims priority to U.S. Provisional App. No.63/329,294, filed on Apr. 8, 2022, titled “Techniques forneuromuscular-signal-based detection of in-air hand gestures for textproduction and modification, and systems, wearable devices, and methodsfor using these techniques,” which is incorporated herein in itsentirety.

TECHNICAL FIELD

The present disclosure relates generally to wearable devices (e.g.,head-worn wearable devices such as augmented-reality glasses andvirtual-reality goggles) and methods for sensing neuromuscular signals,and more particularly to wearable devices configured to detectneuromuscular-based signals corresponding to in-air hand gestures fortext production and modification (e.g., gestures performed by a user'sdigits without contacting any electronic devices, which gestures can beinterpreted to cause modifications to text that was generated based onvoice commands received from a user).

BACKGROUND

Some wearable devices use full-range and space-consuming user movements,such as entire arm, hand, and/or body movements, to detect motor actionsof a user. These devices use the detected motor actions to identify usergestures that correspond to instructions that can be provided as inputsto different computing devices. These full-range movements can bedisruptive and socially unacceptable. Further, to perform the full-rangeuser movements, the user is required to have a minimum amount of spaceavailable (e.g., at least an arm's-width of space) and is required toexpend considerably more energy than is required to operate atouchscreen or handheld device.

For new technologies around text production and modification (editing)using artificial-reality devices (including augmented-reality (AR)glasses and virtual-reality (VR) goggles), these problems aresignificant, as user adoption and use of these new technologies will bediminished (or remain cabined to only certain use cases such as gamingin large open spaces) if the gestures remain socially unacceptable.Moreover, the combined use of multiple input modalities (e.g., sensorsat multiple different wearable devices, such as a smartwatch as well asVR goggles, used to detect different types of gestures and otherinteractions related to text production and modification) to improvetext production and modification requires further exploration to allowfor synergistic and efficient use of these multiple input modalities. Asone example, the ability to use a first input modality to input text(e.g., voice inputs detected via a microphone) and a second inputmodality to modify the inputted text (e.g., gestures that can beperformed by a user without needing to interact with a physical orsimulated/virtual keyboard) requires further exploration. As such, itwould be desirable to address one or more of the above-identifiedissues.

SUMMARY

The systems (wearable devices) and methods described herein address atleast one of the above-mentioned drawbacks by causing the performance ofcommands at a computing device based on detected neuromuscular signalsfrom in-air hand gestures, such as thumb-to-finger-based gestures, whichcan be gestures in which a user either intends to, or actually does,cause their thumb to contact some portion of one of their other digits(or intends to or causes one digit to touch another digit). As willbecome apparent upon reading this disclosure, the in-air hand gesturesdescribed herein are gestures that do not make contact with anelectronic device (such as a smartwatch, generally referred to herein asa wrist-wearable device) and are instead performed in the air. Inparticular, the wearable devices described herein are configured todetect sequences or patterns of neuromuscular signals based on a userperforming (or intending to perform) a particular in-air hand gesture.Each gesture can be associated with a corresponding command at acomputing device (e.g., associations between gestures and respectiveinput commands can be predefined and stored in a memory of the computingdevice and/or the wearable device). The gestures can includethumb-to-finger gestures such as contacting the tip of the thumb to thetip of the index finger. The gestures can also include hand gesturessuch as making a fist or waving the hand. The gestures can also includemovement of a single finger or thumb, such as a thumb swipe gesture oran index finger tap gesture. The gestures can also include doublegestures, such as a double tap gesture, a double pinch gesture, or adouble swipe gesture. The use of double gestures increases the amount ofavailable gestures and also decreases accidently gesture detection. Asone further example, a virtual directional pad (d-pad) in-air gesturecan also be detected via the neuromuscular-signal sensors in someembodiments, which d-pad in-air gesture includes movement on a user'sthumb in either horizontal or vertical directions on top of a portion ofthe user's index finger (e.g., on top of the skin that sits above theproximal phalange portion of the user's index finger).

The wearable devices and methods described herein, after receiving ordetecting the sequence of neuromuscular signals, provide data to thecomputing device that causes the computing device to perform an inputcommand. The systems and methods described herein allow for minimal usermovement to provide the desired input commands at a computing device,which reduces the amount of space required by a user to perform arecognizable gesture (e.g., limiting movement to the user's hand ordigits, which can be moved discreetly), reduces a total amount of energythat a user must expend to perform a gesture and reduces or eliminatesthe use of large awkward movements to perform the gesture. Theseimprovements allow for the wearable device to be designed such that itis comfortable, functional, practical, and socially acceptable forday-to-day use. These improvements are also important for text-basedinput commands, such as typing, editing, and navigating within amessaging application or document-editing application, as other gesturesfor such input commands can be cumbersome and inefficient, especiallywhen used in artificial-reality environments (such as AR and VRenvironments). All this furthers the goal of getting more users to adoptemerging technologies in the AR and VR spaces for more use cases,especially beyond just gaming uses in large open spaces.

Further, the systems described herein can also improve users'interactions with artificial-reality environments and improve useradoption of artificial-reality environments more generally by providinga form factor that is socially acceptable and compact, thereby allowingthe user to wear the device throughout their day and helping to enhancemore of the user's daily activities (and thus making it easier tointeract with such environments in tandem with (as a complement to)everyday life).

Further, as one example as to how the innovative techniques describedherein help to address the multiple input modality problem/explorationoutlined in the background section above, the systems and methodsdescribed herein make use of multiple input modalities in an efficientand synergistic fashion, including by combining text-inputmethodologies, e.g., speech-to-text (STT), with neuromuscular gesturecontrol, such as in-air hand gestures that can be detected by sensingneuromuscular signals traveling through a user's body. A user can enter(and/or switch between) text-input modes, text-modification modes, andtext-display modes using in-air hand gestures detected based on detectedneuromuscular signals (as mentioned earlier, when a user intends toperform one of the in-air hand gestures, a sequence of neuromuscularsignals travels through their body to effectuate the desired motionaction, which sequence of neuromuscular signals can be detected and thenprocessed by the wearable devices (or a device in communicationtherewith) to detect performance of (or an intention to perform) arespective in-air hand gesture). For example, a first type of gesturecan be used to enter the text-input mode. In the text-input mode theuser may enter text via STT. The user can transition to the text-displaymode via another type of gesture or automatically (e.g., “automatically”referring to a system-state change that occurs without the user needingto request that state change via another gesture or other input) afterentering text. A user's input is displayed (e.g., in anartificial-reality environment that can be presented via AR glasses orVR goggles) and the user can enter a modification mode using yet anothergesture. In the modification mode, the user can select a term in thedisplayed text and provide a modification, such as a replacement term orphrase. The user can select the term for modification via one or both ofgaze-based and neuromuscular-signal-based controls. In this way, thetechniques described herein help to create sustained user interactions(e.g., an uninterrupted user interaction with text input andmodification features that does not require clunky and inefficientoperations to switch between input modalities) and improved man-machineinterfaces (e.g., an efficient interface that allows for easy use ofmultiple input modalities).

In accordance with some embodiments, a method is performed on a wearabledevice having memory and one or more processors. The method includes (i)causing display, using a display that is in communication with awearable device, of a plurality of text terms input by a user; (ii)detecting, using data from one or more neuromuscular-signal sensors incommunication with the wearable device, an in-air hand gesture performedby the user while the plurality of text terms are displayed; (iii) inresponse to the in-air hand gesture, enabling a text-modification modethat allows for modifying the plurality of text terms input by the user;and (iv) while the text-modification mode is enabled (a) identifying atarget term of the plurality of text terms, (b) receiving data about avoice input provided by the user for modifying the target term, and (c)causing a modification to the target term in accordance with the voiceinput from the user.

In some embodiments, a computing device (e.g., a wrist-wearable deviceor a head-mounted device or an intermediary device such as a smart phoneor desktop or laptop computer that can be configured to coordinateoperations at the wrist-wearable device and the head-mounted device)includes one or more processors, memory, a display (in some embodiments,the display can be optional, such as for certain example intermediarydevices that can coordinate operations at the wrist-wearable device andthe head-mounted device, and thus have ample processing and powerresources but need not have displays of their own), and one or moreprograms stored in the memory. The programs are configured for executionby the one or more processors. The one or more programs includeinstructions for performing (or causing performance of) any of themethods described herein (e.g., including methods 500 and 600 that aredescribed in detail below).

In some embodiments, a non-transitory computer-readable storage mediumstores one or more programs configured for execution by a computingdevice (e.g., a wrist-wearable device or a head-mounted device or anintermediary device such as a smart phone or desktop or laptop computerthat can be configured to coordinate operations at the wrist-wearabledevice and the head-mounted device) having one or more processors,memory, and a display (in some embodiments, the display can be optional,such as for certain example intermediary devices that can coordinateoperations at the wrist-wearable device and the head-mounted device, andthus have ample processing and power resources but need not havedisplays of their own). The one or more programs include instructionsfor performing (or causing performance of) any of the methods describedherein (e.g., including methods 500 and 600 that are described in detailbelow).

Thus, methods, systems, and computer-readable storage media aredisclosed for neuromuscular-signal-based detection of in-air handgestures for text production and modification. Such methods maycomplement or replace conventional methods for text production andmodification.

The features and advantages described in the specification are notnecessarily all-inclusive and, in particular, some additional featuresand advantages will be apparent to one of ordinary skill in the art inview of the drawings, specification, and claims provided in thisdisclosure. Moreover, it should be noted that the language used in thespecification has been principally selected for readability andinstructional purposes, and has not necessarily been selected todelineate or circumscribe the subject matter described herein.

BRIEF DESCRIPTION OF THE DRAWINGS

So that the present disclosure can be understood in greater detail, amore particular description may be had by reference to the features ofvarious embodiments, some of which are illustrated in the appendeddrawings. The appended drawings, however, merely illustrate pertinentfeatures of the present disclosure and are therefore not necessarily tobe considered limiting, for the description may admit to other effectivefeatures, as the person of skill in this art will appreciate uponreading this disclosure.

FIGS. 1A-1F illustrate an example user scenario with anartificial-reality system (e.g., including at least virtual-reality (VR)goggles and a wrist-wearable device) in accordance with someembodiments.

FIGS. 2A-2G illustrate an example user scenario with anartificial-reality system (e.g., including at least augmented-reality(AR) glasses and a wrist-wearable device) in accordance with someembodiments.

FIGS. 3A-3H illustrate an example user scenario in which in-air handgestures detected via a wearable device are used fordocument-manipulation purposes at a computing device in accordance withsome embodiments.

FIGS. 4A-4F illustrate another example user scenario with anartificial-reality system in accordance with some embodiments.

FIGS. 5A-5D are flow diagrams illustrating an example method formodifying text in accordance with some embodiments.

FIGS. 6A-6C are flow diagrams illustrating an example method forinputting text in accordance with some embodiments.

FIGS. 7A-7B are block diagrams illustrating example artificial-realitysystems in accordance with some embodiments.

FIG. 8A shows example AR glasses (which can be used with someembodiments of the artificial-reality system) in accordance with someembodiments.

FIG. 8B shows example VR goggles (which can be used with someembodiments of the artificial-reality system) in accordance with someembodiments.

FIGS. 9A-9C illustrate example wearable devices in accordance with someembodiments.

In accordance with common practice, the various features illustrated inthe drawings are not necessarily drawn to scale, and like referencenumerals may be used to denote like features throughout thespecification and figures.

DETAILED DESCRIPTION

Numerous details are described herein in order to provide a thoroughunderstanding of the example embodiments illustrated in the accompanyingdrawings. However, some embodiments can be practiced without many of thespecific details, and the scope of the claims is only limited by thosefeatures and aspects specifically recited in the claims. Furthermore,well-known processes, components, and materials have not necessarilybeen described in exhaustive detail so as to avoid obscuring pertinentaspects of the embodiments described herein.

Embodiments of this disclosure may include or be implemented inconjunction with various types or embodiments of artificial-realitysystems. Artificial reality constitutes a form of reality that has beenaltered by virtual objects for presentation to a user. Such artificialreality may include and/or represent virtual reality (VR), augmentedreality (AR), mixed reality (MR), hybrid reality, or some combinationand/or variation of one or more of the these. Artificial-reality contentmay include completely generated content or generated content combinedwith captured (e.g., real-world) content. The artificial-reality contentmay include video, audio, haptic feedback, or some combination thereof,any of which may be presented in a single channel or in multiplechannels (such as stereo video that produces a three-dimensional effectto a viewer). Additionally, in some embodiments artificial reality mayalso be associated with applications, products, accessories, services,or some combination thereof that are used, for example, to createcontent in an artificial reality and/or are otherwise used in (e.g., toperform activities in) an artificial reality.

Artificial-reality systems may be implemented in a variety of differentform factors and configurations. Some artificial-reality systems includea near-eye display (NED), which provides visibility into the real world(e.g., the AR system 820 in FIG. 8A) or that visually immerses a user inan artificial reality (e.g., the VR system 850 in FIG. 8B). While someartificial-reality devices are self-contained systems, otherartificial-reality devices communicate and/or coordinate with externaldevices to provide an artificial-reality experience for a user. Examplesof such external devices include handheld controllers, mobile devices,desktop computers, devices worn by a user (e.g., the wearable device 900in FIG. 9A), devices worn by one or more other users, and/or any othersuitable external system.

FIGS. 1A-1F illustrate an example user scenario with anartificial-reality system 100 (e.g., including at least VR goggles and awrist-wearable device) in accordance with some embodiments. Theartificial-reality system 100 includes a head-mounted display device 102(also referred to as a head-worn wearable device or simply as ahead-mounted or head-worn device, and the head-mounted device is also awearable device since it is worn on the user's head) and awrist-wearable device 104. Other examples of wearable devices includerings, anklets, armbands, neckbands, headbands, and smart clothing(e.g., clothing with integrated sensors and electronics). The user 101in FIG. 1A is viewing a scene with a messenger application 108 beingdisplayed using the head-mounted display device 102. The messengerapplication 108 includes multiple messages between the user 101 and aperson “M.” In the example of FIG. 1A, the user has composed a draftmessage 110 that has not yet been sent to the person “M,” as denoted bythe “Not yet sent” state indicator 112. While the example in FIG. 1A isof an electronic messaging conversation/thread between the user and oneother user (“M”), the skilled artisan will appreciate that thetechniques described herein also apply to group conversations betweenthe user and multiple other users (e.g., “M” and one or more additionalusers). While not shown in FIG. 1A, the skilled artisan will alsoappreciate that information exchanged between the devices 102 and 104can be directly exchanged (e.g., over a wireless communication protocolsuch as BLUETOOTH) or can be indirectly exchanged via an intermediary(e.g., using a smart phone or other computing device to coordinate orotherwise handle the exchange of information between the two devices).

In FIG. 1B, the user 101 performs a gesture 120 (e.g., a thumb and indexfinger pinch gesture) in which one or both of the thumb and index fingerare moved toward one another and eventually make contact in the air andthe gesture is detected by the wrist-wearable device 104. In thedepicted example of FIG. 1B, the thumb makes contact with the distalphalange portion of the user's index finger without making any contactwith either of the devices 102 and 104. In some embodiments, the gestureis detected by processing detected sensor data (which can be processedat the wrist-wearable device 104 or at a device that is in communicationtherewith, which can be sensor data from neuromuscular-signal sensorsthat sense neuromuscular signals traveling through the user's body tocause the motor actions that move the thumb and/or index finger towardone another to make contact in the air). In some embodiments, thewrist-wearable device includes one or more neuromuscular sensors fordetecting user gestures, such as the thumb to index finger pinch gestureof FIG. 1B. In some embodiments, the neuromuscular sensors include oneor more surface electromyography (sEMG) sensors, mechanomyographysensors, and/or sonomyography sensors. Techniques for processingneuromuscular signals are described in commonly owned U.S. PatentPublication No. US 2020/0310539, which is incorporated by referenceherein for all purposes, including for example the techniques shown anddescribed with reference to FIGS. 29-30 in the incorporated publication,which can be applied in one example to process neuromuscular signals toallow for detecting the in-air hand gestures described herein. FIG. 1Bfurther shows the messenger application 108 enabling a text-modificationmode (and also disabling the text-review mode that was shown in FIG. 1A)in response to the user gesture 120, as denoted by the “Editing” stateindicator 122. FIG. 1B also shows a term 124 (“forget”) emphasized inthe draft message 110, e.g., in accordance with a user gaze directedtoward the term 124.

In FIG. 1C, the user 101 performs a gesture 128 (e.g., a thumb swipegesture in which the user moves their thumb in a generally rightwarddirection across skin that is above a proximal phalange portion of theuser's index finger) and the gesture is detected by the wrist-wearabledevice 104. FIG. 1C further shows emphasis in the draft message 110moved to the term 130 (“Sarah”) in accordance with the gesture 128 (ascompared to what was shown in FIG. 1B, the gesture 128 can cause theemphasis to move from “forget” (as was shown in FIG. 1B) to “to” andthen to “pick” and then to “up” before reaching “Sarah”). A speedassociated with the gesture 128 can determine whether the emphasis movesacross these other words or jumps directly to “Sarah” (e.g., if thegesture 128 is performed with a speed below a word-skipping threshold(e.g., a threshold of 50 cm/s, 20 cm/s, or 10 cm/s), then the gesture128 would be interpreted to cause incremental movement of the emphasisacross each word, whereas if the gesture 128 is performed with a speedthat is above the word-skipping threshold, then the gesture 128 would beinterpreted to cause movement of the emphasis directly to a proper nounin the sequence of words). The speed of the gesture 128 can be detectedby processing the detected neuromuscular signals associated withperformance of the gesture 128. In some embodiments, the gesture 128corresponds to a gesture performed using a virtual directional pad(d-pad), which in this example is a swipe that moves in a rightwarddirection over the index finger to move the emphasis in the draftmessage 110 to the right, and other directional movements of the thumbdetected over the skin that sits above the proximal phalange portion ofthe user's index finger would cause corresponding directional changes inthe emphasis as it moves across the terms shown in draft message 110.

In FIG. 1D, the user 101 vocalizes a spoken replacement term 134(“Kira”) for the emphasized term 130 in FIG. 1C and the spokenreplacement term 134 is detected by one or both of the head-mounteddisplay device 102 and the wrist-wearable device 104. In accordance withsome embodiments, the head-mounted display device 102 includes amicrophone 131 to detect speech from the user 101. In accordance withsome embodiments, the wrist-wearable device 104 includes a microphone133 to detect speech from the user 101. FIG. 1D further shows thereplacement term 136 (“Kira”) inserted in the draft message 110 (andalso illustrates that the previously emphasized term “Sarah” ceases tobe displayed and the emphasis is now displayed over the replacement term136) in accordance with the spoken replacement term 134.

In FIG. 1E, the user 101 performs a gesture 140 (e.g., the thumb andindex finger pinch gesture in which one or both of the user's thumb andindex finger are moved to contact one another, e.g., the distal phalangeportion of the thumb is made to contact the distal phalange portion ofthe index finger) and the gesture is detected by the wrist-wearabledevice 104 based at least in part on sensor data. In some embodiments,as is explained in greater detail below in reference to FIG. 7B, thesensor data is data from neuromuscular sensors. In some embodiments,cameras positioned on one or both of the wrist-wearable device and thehead-mounted device can also provide data that is used to help detectthe in-air gestures described herein. FIG. 1E further shows themessenger application 108 disabling the text-modification mode (andswitching back/re-enabling to a text-review mode) in response to theuser gesture 140, as denoted by the “Not yet sent” state indicator 142.In accordance with some embodiments, the draft message 110 in FIG. 1Edoes not have an emphasized term due to the text-modification mode beingdisabled (e.g., terms are not selected or emphasized while thetext-modification mode is disabled, which can include disabling thesensors used for gaze-tracking purposes after an instruction is sentfrom the wrist-wearable device to the head-worn device to disable thesensors used for gaze tracking that are coupled with the head-worndevice, and this disabling feature can help to preserve limitedcomputing and power resources at the head-worn device while also helpingto further a sustained user interaction with the messenger applicationthat gracefully shifts between text-review and text-modification modes).In some embodiments, the gesture 140 is the same as the gesture 120,which means that in these embodiments the same gesture is used to bothenable and then later disable the text-modification mode. In conjunctionwith these embodiments, once the text-modification mode is enabled forthe messaging application, the gesture 120/140 is not used for any otherpurpose, which helps to further a sustained user interaction andimproved man-machine interface as the gesture 120/140, as use of thesame gesture for enabling and disabling the text-modification mode helpsto avoid a situation in which a user unintentionally enables or disablesthe text-modification mode. To further this goal of avoidingunintentional activation or deactivation of the text-modification mode,the gesture 120/140 can also have an associated time component, e.g.,the contact between the index finger and thumb must last for at least agesture-activation time threshold (e.g., a value within the range of10-20 ms) to then cause enabling or disabling of the text-modificationmode. In addition to, or as an alternative to, use of thegesture-activation time threshold, the gesture 120/140 can involve theuser's thumb making contact with a digit other than their index finger(e.g., pinky finger) as that gesture is less likely to be accidentallyperformed as compared to other gestures.

In FIG. 1F, the user 101 performs a gesture 146 (e.g., an index fingerswipe) and the gesture is detected by the wrist-wearable device 104based on sensor data. In some embodiments, the gesture 146 is an indexfinger flick gesture in which the user performs a motor action thatcauses the index finger to move across a medial and/or proximal phalangeportion of the thumb toward a distal phalange portion of the thumbquickly in a flicking action away from the user's body. FIG. 1F furthershows the messenger application 108 causing the sending of the message149 to the person “M” in response to detecting the gesture 146, asdenoted by the “Sent” state indicator 148. In accordance with someembodiments, the message 149 in FIG. 1F is visually distinct from thedraft message 110 in FIG. 1E to denote that it has been sent to theperson “M.” In some embodiments, the gesture 146 is a multipart gesture,such as a double swipe or flick gesture, in which the user performs thegesture 146 twice in succession (e.g., within a short period of timesuch as within 10 milliseconds, 100 milliseconds, or 1 second). In someembodiments, the multipart gesture is a combination of two or moregestures such as a flick-then-pinch gesture, in which the user performsthe gesture 146 followed by a middle finger and thumb pinch gesture(e.g., within a short period of time such as within 10 milliseconds, 100milliseconds, or 1 second). In some embodiments, the gesture 146 is amultipart gesture so as to reduce or prevent accidental sending of draftmessages. In some embodiments, a prompt is displayed (e.g., at thehead-mounted display device 102 or the wrist-wearable device 104) to theuser to allow them to confirm their intention to send the draft message110 before the sending occurs.

FIGS. 2A-2G illustrate another example user scenario with anartificial-reality system 200 (e.g., including at least AR glasses and awrist-wearable device) in accordance with some embodiments. Theartificial-reality system 200 includes AR glasses 202 and thewrist-wearable device 104. The user 101 in FIG. 2A is viewing a scenewith the messenger application 108 displayed using the AR glasses 202(the depicted scene can be superimposed, e.g., using a heads-up displayof the AR glasses 202, on top of physical aspects of the user's reality,such as superimposed on top of a physical table or a wall within theuser's house or office space). The messenger application 108 includesmultiple messages between the user 101 and a person “M.” In the exampleof FIG. 2A, the user is editing a draft message 206, as denoted by the“Editing” state indicator 208.

FIG. 2B shows the user 101 looking at the term 215 (“don't”) and gazetracking is being performed by the AR glasses 202, where the user's gazein the depicted example is denoted by the gaze lines 214. In someembodiments, the gaze tracking is performed using one or moreeye-tracking cameras of the AR glasses 202. FIG. 2B further shows theterm 215 emphasized (e.g., denoted in this example by the box-shapeddashed lines) in accordance with the gaze tracking. In some embodiments,the gaze tracking is enabled at the AR glasses 202 in accordance withthe text-modification mode being enabled. In some embodiments, the gazetracking is disabled in accordance with the text-modification mode beingdisabled. In some embodiments, rather than identify a specific term foremphasis, the gaze tracking can be used to identify a region of text towhich the user's gaze is directed (e.g., multiple terms receive theemphasis rather than a single term). In still other embodiments, gazetracking can be replaced (or supplemented) by use of the d-pad gesturesdescribed earlier in which movement of the user's thumb in variousdirections over the skin that sits above the proximal phalange portionof the user's index finger cause a corresponding directional change tomove the emphasis between terms in the message that is being composed.

FIG. 2C shows the user 101 shifting their gaze to the term 216 (“park”)and gaze tracking being performed by a component (e.g., eye-trackingcamera(s)) associated and/or coupled with the AR glasses 202, denoted bythe gaze lines 214. FIG. 2C further shows the term 216 emphasized (e.g.,denoted in this example by the box-shaped dashed lines) in accordancewith the gaze tracking.

In FIG. 2D, the user 101 performs a gesture 220 (e.g., athumb-and-index-finger pinch gesture, which is analogous to the gesture120 described earlier, so those descriptions apply to the gesture 220 aswell) and the gesture is detected by the wrist-wearable device 104. Inaccordance with some embodiments, the gesture 220 corresponds to aterm-selection operation, and FIG. 2D further shows the emphasized term216 from FIG. 2C selected in accordance with the gesture 220 (e.g.,replaced with the ellipsis in box-shaped dashed lines 222 indicatingthat the system is ready to receive a replacement from the user 101).Thus, in the embodiments illustrated in the FIG. 2 series, the thumb andindex finger pinch gesture corresponds to a different operation than inthe embodiments illustrated in the FIG. 1 series. The thumb and indexfinger pinch gesture is an illustrative example of a gesture. Inembodiments that encompass both the FIG. 1 and FIG. 2 series, a separategesture can be used for the term-selection operation (e.g., an indexfinger tap to the user's palm) to distinguish it from the gesture usedto enter/exit the text-modification mode (e.g., the thumb and indexfinger pinch gesture).

A similar replacement indication can also be presented in the sequencebetween FIGS. 1C and 1D when the user is going through the process ofreplacing the term “Sarah” with the term “Kira.” In some embodiments, inaddition to the term-selection operation causing the selected term tocease being displayed and to display a replacement indication (e.g., theellipsis), the term-selection operation can also cause the gaze tracking(for embodiments in which gaze tracking is utilized) to be temporarilydisabled.

In FIG. 2E, the user 101 says a replacement phrase 230 (“park onFranklin at 1:55 pm”) and the replacement phrase 230 is detected by theAR glasses 202 and/or the wrist-wearable device 104. In accordance withsome embodiments, the AR glasses 202 include a microphone to detectspeech from the user 101. FIG. 2E further shows the replacement phrase232 (“park on Franklin at 1:55 pm”) inserted in the draft message 206 inaccordance with the spoken replacement phrase 230. In the example ofFIGS. 2D and 2E, the selected term represents a first term (“park”) forthe replacement phrase 230. In some embodiments, the selected termrepresents a term not changed in the replacement phrase for theartificial-reality system 200 (e.g., the messenger application 108). Forexample, the selected term may be “park” and the replacement phrase maybe “Franklin Street park.” In some embodiments, the selected term 222represents a term to be replaced in the replacement phrase 230. Forexample, a message may include “pick up Susan” and the selected term maybe “Susan” with the replacement phrase being “pick up Kira.” In someembodiments, the replacement phrase or term is only detected while thegesture 220 is maintained, e.g., the microphone(s) of the AR glasses 202and/or the wrist-wearable device 104 are activated while the gesture ismaintained to allow for detecting the replacement phrase or term, andthe microphone(s) are deactivated once the gesture 220 is released.

FIG. 2F shows the user 101 looking at the term 242 (“Franklin”) and gazetracking being performed by the AR glasses 202, denoted by the gazelines 214. FIG. 2B further shows the term 242 emphasized (e.g., boxed bydashed lines, which can represent any number of emphasis techniquesincluding color changes, highlighting, and/or an increase in text size)in accordance with the gaze tracking. As was mentioned earlier, forembodiments that do not use gaze tracking (e.g., have gaze trackingdisabled or do not have gaze-tracking hardware at all), the user canperform the d-pad gesture to cause directional movements to selectdifferent terms and cause the emphasis to move according to thosedirectional movements.

In FIG. 2G, the user 101 performs a gesture 246 (e.g., a thumb and ringfinger pinch gesture in which one or both of the thumb and ring fingerare moved to contact one another) and the gesture is detected by thewrist-wearable device 104. In accordance with some embodiments, thegesture 246 corresponds to a replacement-menu operation and FIG. 2Gfurther shows replacement terms 252 and 254 displayed for the emphasizedterm 242 in response to the gesture 246. In some embodiments, thereplacement terms are selected for display based on a language model(e.g., a language model executing on the wrist-wearable device 104).While not illustrated, selection of options from among the replacementterms 252 and 254 can be performed by using the d-pad gesture or byusing gaze tracking, or by using a combination of both techniques. As isalso clear from the depicted examples, the gesture 246 is a differentin-air hand gesture as compared to the gesture 220 (described above inreference to FIG. 2D), so the gesture 220 can be referred to as a firstin-air hand gesture and the gesture 246 can be referred to as a secondin-air hand gesture that is distinct from the first in-air hand gesture.The illustrated example gestures 220 and 246 are examples, and otherin-air hand gestures can also be suitable while still ensuring that thetwo in-air hand gestures are distinct from one another to ensuresustained user interactions.

The examples of the sequences shown in the FIGS. 1 and 2 series havefocused on use of a messaging application, but the techniques describedherein have a broader applicability beyond just messaging applications.For instance, the techniques described herein apply to any applicationin which text needs to be selected and modified, includingdocument-editing applications. The FIG. 3 sequence, which will bediscussed next, provides a more specific example of using thesetechniques for document-editing applications. More specifically, FIGS.3A-3H illustrate an example user scenario in which in-air hand gesturesdetected via a wearable device are used for document-manipulationpurposes at a computing device in accordance with some embodiments. FIG.3A shows the user 101 with the wrist-wearable device 104 and a display302 in communication (either a direct wired or wireless communicationlink between the two devices or one in which an intermediary device isused to communicably connect the two devices) with the wrist-wearabledevice 104. FIG. 3A further shows a document-editing application (e.g.,in the illustrated example, the document-edition application is aword-processing application 301) displaying a document 303 on thedisplay 302. FIG. 3A also shows a selected term 306 (denoted by thedashed-line box around it) in the document 303 (the term can be selectedin accordance with any of the techniques discussed earlier in referenceto the sequences in the series of FIGS. 1 and 2 ) and an actions menu308. The actions menu 308 includes a plurality of actions 310, includingan action 310-1 to delete the selected term 306 and an action 310-3 toopen a context menu. In some embodiments, the actions menu 308 isdisplayed automatically (e.g., without requiring a specific user inputto activate display), for example, is displayed continuously ordisplayed after a set amount of time from receiving a user input (e.g.,1 second, 5 seconds, or 20 seconds). In some embodiments, the actionsmenu 308 is displayed in response to detection of a user gesture, suchas a middle finger to palm tap gesture (where the user moves theirmiddle finger inward to contact a portion of the user's palm). In someembodiments, the actions menu 308 is displayed in response to a voicecommand or other type of user input. In some embodiments, whether theactions menu 308 is displayed is dictated by a user setting (e.g., auser setting associated with the word-processing application 301 and/orthe wrist-wearable device 104).

In accordance with some embodiments, each action 310 in the actions menu308 includes an indication of a corresponding gesture to be performed bythe user 101 to cause performance of a respective action. For example,the delete action 310-1 is caused to be performed after detection of afist gesture (e.g., a gesture in which the user moves all of theirdigits to create a fist with their hand) and the context menu action310-3 is caused to be performed after detection of an air tap gesture(e.g., a gesture in which one of the user's digits is moved in agenerally downward direction to tap within free space). In accordancewith some embodiments, the word-processing application 301 is in atext-modification mode (which can be activated in accordance with any ofthe techniques described above in reference to the FIGS. 1 and 2 series)as denoted by the emphasis around selected term 306. Display ofavailable gestures and their associations with particular actions canalso occur at any time while the text-modification mode is activated,and this applies to the enabled text-modification modes depicted in theother figure sequences as well (e.g., with the messaging application,indications of available in-air hand gesture options can be presented tothe user, which helps to assist with user adoption and learning of a newgesture space, thereby furthering the ability of users to have asustained user interaction).

In FIG. 3B, the user 101 performs a gesture 320 (e.g., an index fingerair tap gesture) and the gesture is detected by the wrist-wearabledevice 104. As shown in FIG. 3A, the gesture 320 corresponds to theaction 310-3, so detection of the air tap shown in FIG. 3B causesopening of the context menu 322. Accordingly, in response to detecting arespective in-air hand gesture (in this example, the air tap of FIG. 3B)that causes performance of a respective action (in this example, openinga context menu), FIG. 3B shows performance of that respective action(e.g., opening a context menu 322 including a plurality of options 324,including a replacement option 324-1 and a capitalization option 324-4).In accordance with some embodiments, the context menu 322 in FIG. 3Bincludes options that are appropriately selected based on the selectedterm 306 and the context surrounding it (e.g., terms near the selectedterm). In some embodiments, the user can select an option 324 via gazetracking and/or d-pad thumb movements (e.g., as described previouslywith respect to FIG. 1C). In some embodiments, the user can activate theselected option by performing a corresponding gesture (e.g., repeatingthe index finger air tap gesture 320 or performing a middle finger airtap gesture).

In FIG. 3C, the user 101 performs a gesture 330 (e.g., a thumb andmiddle finger pinch gesture) and the gesture is detected by thewrist-wearable device 104. In accordance with some embodiments, thegesture 330 corresponds to a close operation and accordingly the contextmenu 322 from FIG. 3B is closed in FIG. 3C. In some embodiments, thegesture 330 is a state-agnostic gesture (e.g., performs a closeoperation regardless of the active state of the word-processingapplication 301).

In FIG. 3D, the user 101 performs a gesture 340 (e.g., a thumb swipegesture that moves directionally on top of skin that is over a proximalphalange portion of the user's index finger) and the gesture is detectedby the wrist-wearable device 104. FIG. 3D further shows emphasis in thedocument 303 moved to a new selected term 342 (“enim”) in accordancewith directional movement indicated by the gesture 340. In someembodiments, the gesture 340 corresponds to a gesture performed using avirtual directional pad (d-pad) and is a down swipe (e.g., a swipe ofthe user's thumb that moves in a generally downward direction over theskin that is over the proximal phalange portion of the user's indexfinger such that the thumb is moved toward the user's body) to move theemphasis in the document 303 down from the term 306 in FIG. 3C to theterm 342 in FIG. 3D. As explained previously, a speed associated withthe thumb swipe gesture can be used to determine whether the emphasismoves gradually between different intervening terms or whether theemphasis jumps to the new selected term 342 without emphasizing anyintervening terms.

In FIG. 3E, the user 101 performs a gesture 350 (e.g., afist/fist-closure gesture) and the gesture is detected by thewrist-wearable device 104. In accordance with some embodiments, thegesture 350 corresponds to a “delete” operation and accordingly theemphasized term 342 in FIG. 3D is deleted in FIG. 3E. FIG. 3E furthershows a term adjacent to the deleted term 342 being selected as the nextselected term 352 now that the new selected term 342 has been deleted.In some embodiments, detection of the gesture associated with the“delete” operation also causes the system to exit the text-modificationmode, such that no term is selected as the next selected term andinstead the emphasis is ceased to be displayed and the system returns toa text-review mode.

In FIG. 3F, the user 101 performs a gesture 356 (e.g., a thumb and ringfinger pinch gesture) and the gesture is detected by the wrist-wearabledevice 104. In accordance with some embodiments, the gesture 356corresponds to an operation for opening a modifier menu, and accordinglythe modifier menu 357 is displayed in FIG. 3F. As with the othergestures shown in the figures, the thumb and ring finger pinch gestureshown in FIG. 3F is an illustrative example gesture for opening amodifier menu, and other gestures can be used instead. In embodimentsthat include functionality for opening multiple menus (e.g., themodifier menu 357, the actions menu 308, and/or the context menu 322), adistinct gesture can be assigned to each menu so as to avoid userconfusion and unintentional activations. For example, a pinch gesturecan correspond to opening the actions menu 308, an air tap gesture cancorrespond to opening the context menu 322, and a palm tap gesture cancorrespond to opening the actions menu 308. In accordance with someembodiments, the modifier menu 357 in FIG. 3F includes a plurality ofmodification options 358, including an option 358-1 to toggle bold textand an option 358-3 to toggle italicized text.

In FIG. 3G, the user 101 performs the gesture 330 (e.g., the thumb andmiddle finger pinch gesture) and the gesture is detected by thewrist-wearable device 104. In accordance with some embodiments, thegesture 330 corresponds to the “close” operation and accordingly themodifier menu 357 from FIG. 3F is closed in FIG. 3G. As was previouslymentioned, the gesture to activate the close operation can becontext-agnostic such that the same in-air hand gesture can be used toclose multiple different types of user interface elements, including themodifier menu 357 and the context menu 322.

In FIG. 3H, the user 101 performs a gesture 360 (e.g., a thumb and indexfinger pinch gesture) and the gesture is detected by the wrist-wearabledevice 104. FIG. 3H further shows the word-processing application 301disabling the text-modification mode in response to the user gesture360, as illustrated by the lack of a selected term. FIG. 3H furthershows the actions menu 308 with a plurality of actions 364 (the actionsmenu 308 can be automatically, and in the absence of a specific userrequest, opened after the text-modification mode is exited out of whenthe user is interacting with a document-editing application). In someembodiments, the actions menu 308 is displayed in accordance with adetermination that the user 101 is likely finished with thedocument-editing application, which can be determined based on past userinteractions with the document-editing application. In accordance withsome embodiments, the plurality of actions 364 in FIG. 3H is differentfrom the plurality of actions in the actions menu 308 in FIG. 3A due tothe word-processing application 301 being in a different mode (e.g.,text-modification mode being enabled in FIG. 3A and disabled in FIG.3H). The plurality of actions 364 includes a save-document action 364-1and an exit-application action 364-3.

FIGS. 4A-4F illustrate another example user scenario with theartificial-reality system 100 in accordance with some embodiments. Theuser 101 in FIG. 4A is viewing a scene with the messenger application108 being displayed using the head-mounted display device 102. Themessenger application 108 includes multiple messages between the user101 and a person “M.” FIG. 4A also shows a new message dialog 401,including an indication of a corresponding gesture (e.g., thumb andindex finger pinch gesture) for activating the new message operation.

In FIG. 4B, the user 101 performs a gesture 408 (e.g., a thumb and indexfinger pinch gesture) and the gesture is detected by the wrist-wearabledevice 104. FIG. 4B further shows the messenger application 108 startinga new message 402 with status message 404 indicating that a microphoneis active and awaiting voice inputs from the user while the gesture 408is held. In some embodiments, one or more of a microphone on thewrist-wearable device 104 and a microphone on the head-mounted displaydevice 102 is activated in accordance with the gesture 408. Thus, in theembodiments illustrated in the FIG. 4 series, the thumb and index fingerpinch gesture corresponds to a different operation than in theembodiments illustrated in the FIG. 1 series. The thumb and index fingerpinch gesture is an illustrative example of a gesture. In embodimentsthat encompass both the FIG. 1 and FIG. 4 series, a separate gesture canbe used for the microphone activation operation (e.g., an index fingertap to the user's palm) to distinguish it from the gesture used toenter/exit the text-modification mode (e.g., the thumb and index fingerpinch gesture). In some embodiments, a gesture intensity is used todistinguish two gestures. For example, a pinch gesture with an intensitybelow a threshold intensity corresponds to a microphone activationoperation and a pinch gesture with an intensity above the intensitythreshold corresponds to a mode-switch operation. In some embodiments,another aspect of the gesture is used to distinguish gestures, such as aduration, speed, direction, or location of the gesture. For example, aquick pinch gesture (e.g., a pinch that has a duration of less than 20milliseconds or 10 milliseconds) corresponds to a first operation and aslow pinch gesture (e.g., a pinch that has a duration of more than 20milliseconds or 10 milliseconds) corresponds to a second operation.

In FIG. 4C, the user 101 provides voice inputs 409 (“Don't forget topick up”) for the new message 402 while holding the gesture 408. Inaccordance with some embodiments, the head-mounted display device 102includes a microphone 414 to detect the voice inputs from the user 101.In accordance with some embodiments, the wrist-wearable device 104includes a microphone 412 to detect voice inputs from the user 101. FIG.4C further shows the text 410 corresponding to the voice inputs 409 inthe new message 402 and a status message 411 indicating that voiceinputs have been received and are being converted to text.

In FIG. 4D, the user 101 continues providing voice inputs with voiceinputs 420 (“Kira at 2 pm stop”) for the new message 402 while holdingthe gesture 408. FIG. 4D further shows the text 422 corresponding to thevoice inputs 420 in the new message 402 and a status message 411indicating that voice inputs have been received and are being convertedto text.

In FIG. 4E, the user 101 has released the gesture 408 and the release ofthe gesture is detected by the wrist-wearable device 104. FIG. 4Efurther shows the messenger application 108 with a draft message 433with status message 434 indicating that the microphone is deactivated(in accordance with the gesture 408 being released) and the message hasnot yet been sent. In some embodiments, the gesture is a toggle-typegesture (rather than a hold-type gesture), and the microphone isactivated the first time the gesture is performed and is deactivated thesecond time the gesture is performed.

In FIG. 4F, the user 101 performs a gesture 440 (e.g., a fist gesture)and the gesture is detected by the wrist-wearable device 104. Inaccordance with some embodiments, the gesture 440 corresponds to adelete operation and accordingly the last term in the message 433(“stop”) in FIG. 4E is deleted in FIG. 4F. Multiple sequentiallyexecuted gestures 440 can also be provided and would then cause, in theillustrated example, deletion of additional terms. In some embodiments,the user 101 performs a gesture (e.g., a wrist-flick gesture where theuser moves their wrist outward (or inward) with a speed above athreshold (e.g., a threshold of 50 cm/s or 100 cm/s)) that correspondsto an undo command and accordingly the last performed operation isundone.

Although the user scenarios described previously with respect to theseries of FIGS. 1, 2, 3, and 4 describe operations being performed bythe wrist-wearable device 104 and head-worn devices 102 and 202, in someembodiments at least a subset of the operations are performed by anintermediary device, such as a smart phone or personal computer, that isin communication with the wearable devices. For example, detection ofspeech from the user 101 in FIG. 1D is optionally detected using amicrophone of the intermediary device. In some embodiments, thewrist-wearable device 104 and the head-worn devices 102 and 202communicate with one another via the intermediary device (e.g., each iscommunicatively coupled to the intermediary device and the intermediarydevice manages interactions between the devices). As another example,the wrist-wearable device 104 can detect the gesture 408 shown in FIG.4B and indicate the detection to the intermediary device. In thisexample, the intermediary device receives the indication and instructsthe head-mounted display device 102 to enable the microphone 414.Examples of intermediary devices can include the computing devices 724described with reference to FIG. 7A and the computer system 772described with in reference to FIG. 7B. In some embodiments, data fromsensors on multiple devices are combined (e.g., at the intermediarydevice) to detect an in-air gesture. For example, data from one or moreoptical sensors of a head-worn device (e.g., the head-mounted displaydevice 102) can be combined with EMG and/or inertial measurement unit(IMU) data from a wrist-worn device (e.g., the wrist-wearable device104) to identify a swipe gesture at a location that corresponds to afirst scroll bar of a user interface rather than a second scroll bardisplayed at a separate location.

Additionally, although the user scenarios described with respect to theseries of FIGS. 1, 2, 3, and 4 are described as separate sequences, insome embodiments the user scenarios are combined with one another. Forexample, the sequence described with respect to FIGS. 4A-4F occursbefore (or after) the sequence described with respect to FIGS. 1A-1F.The sequence described with respect to FIGS. 2A-2G is optionallyperformed with the artificial-reality system 100 and combined with theaspects discussed with respect to the series of FIG. 1 or 4 (or thesequences and aspects of FIGS. 1 and 4 are performed with theartificial-reality system 200). Similarly, the sequence described withrespect to FIGS. 3A-3H is optionally performed with theartificial-reality system 100 or the artificial-reality system 200 andcombined with aspects discussed with respect to the series of any ofFIG. 1, 2 , or 4 (or the sequences and aspects of FIGS. 1, 2, and 4 areperformed with a system that includes the display 302 and thewrist-wearable device 104 shown in FIGS. 3A-3H).

The user scenarios described with respect to the series of FIGS. 1, 2,and 4 involved an example messenger application (messenger application108). However, the sequences, gestures, actions, and operations can beused in conjunction with other types of applications, such asweb-browsing, note-taking, social media, word processing, data-entry,programming, and the like. Similarly, the user scenario described withrespect to the FIG. 3 series involved an example document-editingapplication (e.g., the word-processing application 301). However, thesequences, gestures, actions, and operations can also be used inconjunction with other types of applications, such as web-browsing,note-taking, social media, messaging, data-entry, programming, and thelike.

FIGS. 5A-5D are flow diagrams illustrating a method 500 for modifyingtext in accordance with some embodiments. The method 500 is performed ata computing system (e.g., a computing device 724 in FIG. 7A) having oneor more processors and memory. In some embodiments, the memory storesone or more programs configured for execution by the one or moreprocessors. At least some of the operations shown in FIGS. 5A-5Dcorrespond to instructions stored in a computer memory orcomputer-readable storage medium (e.g., the memory 778 of the computersystem 772 or the memory 756 of the accessory device 752). In someembodiments, the computing system is a wearable device such as thewrist-wearable device 104 or the head-mounted display device 102.

In some embodiments, the wearable device detects (502), using data fromone or more neuromuscular-signal sensors (e.g., the sensors 716 of FIG.7A), a first in-air hand gesture (e.g., a thumb and middle finger pinchgesture) performed by a user. In response to the first in-air handgesture, the wearable device enables an input mode. While in the inputmode, the wearable device receives data about the plurality of textterms input by the user and causes the display of each of the pluralityof text terms as the data is received. For example, FIGS. 4B-4Dillustrate the user 101 providing inputs while holding the gesture 408.FIGS. 4B-4D further show text 410 and 422 displayed in the messengerapplication 108 in response to the provided inputs.

In some embodiments, the plurality of text terms are received (504) viavoice inputs provided by the user. For example, in FIG. 4C the userprovides voice inputs 409 that are converted to the text 410.

The wearable device causes (506) display, using a display that is incommunication with the wearable device (e.g., a display associated withVR goggles or AR glasses), of a plurality of text terms input by a user.For example, FIG. 1A shows a draft message 110 displayed to the user 101via the head-mounted display device 102.

In some embodiments, the plurality of text terms input by the user arecaused (508) to be displayed on a display of the wearable device. Forexample, the plurality of text terms are displayed on the electronicdisplay 718 of the wearable device 702. As another example, theplurality of text terms are displayed on the display screen 901 of thewearable device 900.

In some embodiments, the wearable device is (510) a wrist-wearabledevice that is configured to send instructions to a head-worn wearabledevice that includes the display. For example, the wearable device isthe wrist-wearable device 104 in FIG. 1A, and the head-worn wearabledevice is the head-mounted display device 102.

In some embodiments, the wearable device causes (512) display of arepresentation of one or more available gesture commands. For example,FIG. 3A shows an actions menu 308 with a plurality of actions 310 andcorresponding gesture commands.

In some embodiments, the one or more available gesture commands are(514) based on an operational mode of the wearable device and therepresentation is updated as the operational mode of the wearable devicechanges. For example, FIG. 3A shows the plurality of actions 310associated with a text-modification mode of the word-processingapplication 301 and FIG. 3H shows the plurality of actions 364associated with a non-text-modification mode (e.g., a text-viewing mode)of the word-processing application 301. The one or more availablegesture commands can be continually updated, which helps to train andmake users familiar with a new in-air hand gesture space. Over time(e.g., once the system recognizes that the user has learned the newin-air hand gesture space), the system can cease to display some or allof the one or more available gesture commands.

The wearable device detects (516), using data from one or moreneuromuscular-signal sensors in communication with the wearable device,a second in-air hand gesture (e.g., a thumb and index finger pinchgesture) performed by the user while the plurality of text terms aredisplayed. For example, FIG. 1B shows the user 101 performing thegesture 120 (e.g., detected by the wrist-wearable device 104).

The wearable device enables (518) a text-modification mode that allowsfor modifying the plurality of text terms input by the user in responseto the second in-air hand gesture. For example, FIG. 1B further showsthe messenger application 108 in a text-modification mode (denoted bythe state indicator 122) in response to the gesture 120.

While the text-modification mode is enabled (520), the wearable deviceidentifies (522) a target term of the plurality of text terms. Forexample, FIG. 1C shows the user performing a thumb swipe gesture toemphasize the term 130.

In some embodiments, the target term is identified (522) based on a gazeof the user. For example, FIG. 2C shows the term 216 being identifiedbased on the gaze lines 214 of the user 101.

In some embodiments, the target term is identified (524) in accordancewith detection, using data from the one or more neuromuscular-signalsensors, of a third in-air hand gesture (e.g., a thumb and index fingerpinch gesture) performed by the user. For example, FIG. 2D shows theuser performing the gesture 220 and selection of the emphasized term 216from FIG. 2C (e.g., the box-shaped dashed lines 222).

In some embodiments, the target term is identified (526) based on adetermination that a term type of the target term matches aclassification of the voice input from the user. In some embodiments,term types include a time type, a day type, a month type, a locationtype, a proper noun type, a number type, and a punctuation type. Forexample, the user says a time (e.g., “2:00 pm”) and a target term havinga time type is identified (e.g., “3:00 pm”). In some embodiments, theterm types include types of words, such as verbs, adverbs, nouns,adjectives, etc.

The wearable device receives (530) data about a voice input provided bythe user for modifying the target term. For example, the wearable devicereceives the data via the microphone 133. As another example, thewearable device receives the data from another device in communicationwith the wearable device (e.g., the head-mounted display device 102).

The wearable device causes (532) a modification to the target term inaccordance with the voice input from the user. For example, FIG. 1Dshows the user saying the replacement term 134 and the replacement term136 (“Kira”) being inserted in the draft message 110 in accordance withthe spoken replacement term 134.

In some embodiments, the wearable device tracks (534) a user gaze. Thewearable device causes the target term to be emphasized on the displayin accordance with the user gaze. For example, FIG. 2B shows the ARglasses 202 tracking the gaze of the user 101 (e.g., as denoted by thegaze lines 214). FIG. 2B further shows the term 215 emphasized inaccordance with the user gaze tracking. The wearable device identifiesthe target term by detecting, using data from the one or moreneuromuscular-signal sensors, a fourth in-air hand gesture while thetarget term is emphasized. For example, FIG. 2D shows the emphasizedterm 216 from FIG. 2C selected in accordance with the gesture 220 andthe gaze-tracking lines 214.

In some embodiments, the wearable device causes (536) a first term ofthe plurality of text terms to be emphasized on the display. The firstterm appears before the target term in the plurality of text terms. Thevoice input is received while the first term is emphasized on thedisplay. The voice input is determined so as to modify a phrase thatincludes the first term and the target term. For example, FIGS. 2D and2E show the user selecting a term (e.g., the first term) and saying areplacement phrase 230. FIG. 2E further shows the replacement phrase 232(“park on Franklin at 1:55 pm”) inserted in the draft message 206 inaccordance with the spoken replacement phrase 230. In this example, thetarget term could be “2:00 pm,” which is replaced with “1:55 pm” inaccordance with the user's replacement phrase 230.

In some embodiments, the gesture to enable the text-modification mode isa held gesture (e.g., a pinch gesture with a duration of at least 20milliseconds, 50 milliseconds, or 500 milliseconds) wheretext-modification mode is only enabled while the gesture is held. Insome embodiments, the gesture to select a term is a forceful (deep)press of the held gesture. For example, the held gesture is an indexfinger and thumb pinch gesture having an intensity below a presetthreshold (e.g., 50 grams, 100 grams, or 200 grams) and the forcefulpress is an increase in intensity of the pinch gesture above the presetthreshold.

In some embodiments, the wearable device detects (538), using data fromthe one or more neuromuscular-signal sensors, a fifth in-air handgesture (e.g., a thumb and pinky finger pinch gesture) performed by theuser. In response to the fifth in-air hand gesture, the wearable devicecauses display of a context menu in proximity to the plurality of textterms. For example, FIG. 3B shows the user 101 performing the gesture320 and the context menu 322 being displayed in response. In someembodiments, the context menu includes options to copy, cut, and/orpaste text. In some embodiments, the context menu corresponds to adouble gesture (e.g., a double swipe, double pinch, or double tapgesture).

In some embodiments, the wearable device detects (540), using data fromthe one or more neuromuscular-signal sensors, a sixth in-air handgesture (e.g., a thumb to palm tap gesture) performed by the user. Inresponse to the sixth in-air hand gesture, the wearable device causesdisplay of one or more modifiers. For example, FIG. 3F shows the user101 performing the gesture 356 and the modifiers menu 357 beingdisplayed in response. In some embodiments, the sixth in-air handgesture corresponds to a modifier command. For example, a forceful(deep) middle finger and thumb press may correspond to a “Shift” keytoggle. As another example, a pinky to palm tap may correspond to a“Ctrl” key toggle.

In some embodiments, the wearable device detects (542), using data fromthe one or more neuromuscular-signal sensors, a seventh in-air handgesture (e.g., a hand flick gesture) performed by the user. In responseto the seventh in-air hand gesture, the wearable device causes thedeletion of one or more terms of the plurality of text terms from beingdisplayed on the display. For example, FIG. 3E shows the user 101performing the gesture 350 and the emphasized term 342 in FIG. 3D havingbeen deleted in FIG. 3E in response to the gesture 350.

In some embodiments, the wearable device detects (544), using data fromthe one or more neuromuscular-signal sensors, an eighth in-air handgesture performed by the user. In response to the eighth in-air handgesture, the wearable device exits the text-modification mode. In someembodiments, the eighth in-air hand gesture is the same as the secondin-air hand gesture. For example, the second in-air hand gesture togglesthe text-modification mode on, and the eighth in-air hand gesturetoggles the text-modification mode off. As an example, FIG. 1B shows theuser 101 performing the gesture 120 and a text-modification mode beingenabled for the messenger application 108 and FIG. 1E shows the user 101performing the gesture 140 and the text-modification mode being disabledfor the messenger application 108. In some embodiments, disabling thetext-modification mode includes transitioning to a text-review mode(also sometimes called a text-display mode). In some embodiments, thetext-review mode corresponds to an input mode, where new inputs from theuser 101 are appended to the displayed text.

In some embodiments, the wearable device causes (546) a first term ofthe plurality of text terms to be emphasized on the display. Whilecausing the first term to be emphasized, the wearable device detects,using data from the one or more neuromuscular-signal sensors, a ninthin-air hand gesture performed by the user. In response to the ninthin-air hand gesture, the wearable device displays a menu with one ormore replacement options for the first term, where the one or morereplacement options are obtained from a speech model. For example, FIG.2G shows the user 101 performing the gesture 246 and replacement terms252 and 254 displayed in response to the gesture 246. In someembodiments, the replacement options include one or more of synonyms,homonyms, or homophones for the first term. In some embodiments, thefirst term corresponds to a first speech-to-text translation for a voiceinput from the user and the replacement options include one or moresecondary speech-to-text translations for the voice input.

In some embodiments, the wearable device disables (548) user gazetracking while the text-modification mode is not enabled. In someembodiments, user gaze tracking is disabled while in one or morenon-text-modification modes (e.g., a text-input mode or text-displaymode). In some embodiments, user gaze tracking is only enabled while thetext-modification mode is enabled.

In some embodiments, in response to detecting a tenth in-air handgesture (e.g., a pinch and shake gesture) using data from the one ormore neuromuscular-signal sensors, the wearable device causes (550) theplurality of text terms, including the target term after it has beenmodified, to be sent to one or more other users via a messagingapplication. For example, FIG. 1F shows the user performing the gesture146 and the draft message 110 from FIG. 1E being sent to the person “M”in response to the gesture 146. An example pinch and shake gestureincludes a thumb and pinky finger pinch with a concurrent wrist shake(e.g., the pinch is held for at least a threshold duration such as 100milliseconds and the user shakes their wrist while holding the pinchgesture).

FIGS. 6A-6C are flow diagrams illustrating a method 600 for inputtingtext in accordance with some embodiments. The method 600 is performed ata computing system (e.g., a computing device 724 in FIG. 7A) having oneor more processors and memory. In some embodiments, the memory storesone or more programs configured for execution by the one or moreprocessors. At least some of the operations shown in FIGS. 6A-6Ccorrespond to instructions stored in a computer memory orcomputer-readable storage medium (e.g., the memory 778 of the computersystem 772 or the memory 756 of the accessory device 752). In someembodiments, the computing system is, or includes, a wearable devicesuch as the wrist-wearable device 104 or the head-mounted display device102. In some embodiments, the computing system is, or includes, anintermediary device such as a smart phone.

While data associated with a messaging application is displayed on adisplay that is in communication with a wearable device, the wearabledevice detects (602), using data from one or more neuromuscular-signalsensors in communication with the wearable device, a first in-air handgesture performed by the user. For example, FIG. 4B shows the user 101performing the gesture 408 while the messenger application 108 isdisplayed.

In some embodiments, the wearable device is (604) a wrist-wearabledevice (e.g., the wrist-wearable device 104) that is configured to sendinstructions to a head-worn wearable device that includes the display(e.g., the head-mounted display device 102).

In some embodiments, the wearable device is (606) a head-mounted device(e.g., the AR glasses 202) that is configured to communicate with one ormore additional wearable devices (e.g., the wrist-wearable device 104).

In response to the first in-air hand gesture, the wearable deviceenables (608) a microphone to capture audio for use in conjunction withthe messaging application. For example, FIG. 4B further shows microphone414 and microphone 412 enabled in response to the gesture 408, asdenoted by the status message 404.

The wearable device receives (610) data about a voice input from theuser via the microphone. For example, FIG. 4C shows the user 101providing voice inputs 409 received by at least one of the microphone414 and the microphone 412. In some embodiments, the wearable devicereceives the data via another device (e.g., a head-worn device or anintermediary device). In some embodiments, an intermediary devicereceives the data about the voice input.

The wearable device converts (612) the voice input to a plurality oftext terms. For example, FIG. 4C further shows the text 410corresponding to the voice inputs 409 in the new message 402 and thestatus message 411 indicating that voice inputs have been received andare being converted to text. In some embodiments, an intermediary deviceconverts the voice input.

The wearable device causes (614) display, within the messagingapplication on the display, of the plurality of text terms to the user.For example, FIG. 4D shows the text 422 corresponding to the voiceinputs 409 and 420 displayed to the user 101 in the messengerapplication 108. In some embodiments, an intermediary device causes thedisplay of the plurality of text terms.

In some embodiments, the wearable device enables (616) an input mode inresponse to the first in-air hand gesture. While the input mode isenabled and after converting the voice input to the plurality of textterms, the wearable device detects, using data from the one or moreneuromuscular-signal sensors, a second in-air hand gesture performed bythe user. In response to the second in-air hand gesture, the wearabledevice causes a deletion of a term of the plurality of text terms. Forexample, FIG. 4F shows the user 101 performing the gesture 440 andcorresponding deletion of the last term in the message 433 (“stop”) inFIG. 4E.

In some embodiments, the first in-air gesture is (618) a gesture duringwhich the user's thumb is held against a user's digit for at least apredetermined period (e.g., a period of time that is at or between 10-20ms). The wearable device disables the microphone in response todetecting the release of the first in-air hand gesture. For example,FIGS. 4B-4D show the user holding the gesture 408 while providing voiceinputs 409 and 420 and FIG. 4E shows the user releasing the gesture 408and the microphone being disabled in response (as denoted by the statusmessage 434).

In some embodiments, the first in-air hand gesture is (620) a togglegesture that is detected at a first point in time. The wearable devicedisables the microphone in response to a subsequent detection of thefirst in-air hand gesture at a second point in time that is after thefirst point in time. For example, a first thumb and middle finger pinchgesture toggles the microphone on and a subsequent second thumb andmiddle finger pinch gesture toggles the microphone off. An example ofthis is shown and described in reference to FIGS. 4A-4D.

In some embodiments, the wearable device detects (622), using data fromthe one or more neuromuscular-signal sensors, a third in-air handgesture. In response to the third in-air hand gesture, the wearabledevice enables a text-modification mode. While the text-modificationmode is enabled, the wearable device tracks the gaze of the user withrespect to the plurality of text terms displayed within the messagingapplication. The wearable device causes a term of the plurality of textterms to be emphasized in accordance with the tracking of the gaze ofthe user. For example, FIG. 2C shows the term 216 being identified basedon the gaze lines 214 of the user 101. In some embodiments, an initialterm is emphasized in accordance with gaze tracking. In someembodiments, the user may emphasize a different term by performing anavigation gesture (e.g., a thumb d-pad swipe gesture). In someembodiments, gaze tracking is performed while a user holds acorresponding gesture (e.g., while the user holds a thumb and indexfinger forceful (deep) pinch gesture). In some embodiments, the gazetracking emphasizes the initial term in accordance with the user lookingat the initial term when releasing the held gesture.

In some embodiments, the wearable device disables (624) user gazetracking while the text-modification mode is not enabled. In someembodiments, user gaze tracking is disabled while in one or morenon-text-modification modes (e.g., a text-input mode or text-displaymode). In some embodiments, user gaze tracking is only enabled while thetext-modification mode is enabled.

In some embodiments, while a text-modification mode is enabled, thewearable device detects (626), using data from the one or moreneuromuscular-signal sensors, a fourth in-air hand gesture performed bythe user. In response to the fourth in-air hand gesture, the wearabledevice causes a deletion of the emphasized term. For example, FIG. 3Eshows the user 101 performing the gesture 350 and the emphasized term342 in FIG. 3D having been deleted in FIG. 3E in response to the gesture350.

In some embodiments, the wearable device detects (628), using data fromthe one or more neuromuscular-signal sensors, a fifth in-air handgesture. In response to detecting the fifth in-air hand gesture, thewearable device causes the plurality of text terms to be sent to one ormore other users via the messaging application. For example, FIG. 1Fshows the user performing the gesture 146 and the draft message 110 fromFIG. 1E being sent to the person “M” in response to the gesture 146.

As one of skill in the art will appreciate, aspects of the method 600can be combined and/or replaced with aspects of the method 500. Forexample, the method 600 can be performed prior to (or after) the method500. The method 600 can include the operations of method 500, e.g.,operations from method 500 can be performed after operation 614 andprior to operation 628 (so as to edit a message prior to it being sentto another user). As another example, the operation 506 can be replacedwith the operation 614.

Having thus described example sequences and methods of operation thatmake use of the example sequences, attention will now be directed tosystem-level depictions of hardware and software on which (or withwhich) the methods can be implemented.

Example System-Level Block Diagrams

FIGS. 7A-7B are block diagrams illustrating example components used withartificial-reality systems in accordance with some embodiments. FIG. 7Ais a block diagram illustrating an artificial-reality system 700 inaccordance with some embodiments. While some example features areillustrated, various other features have not been illustrated for thesake of brevity and so as not to obscure pertinent aspects of theexample embodiments disclosed herein. To that end, as a non-limitingexample, the system 700 includes one or more wearable devices 702, whichcan be used in conjunction with one or more computing devices 724. Insome embodiments, the system 700 provides the functionality of avirtual-reality (VR) device, an augmented-reality (AR) device, amixed-reality device, hybrid-reality device, or a combination thereof.In some embodiments, the system 700 provides the functionality of a userinterface and/or one or more user applications (e.g., games, wordprocessors, messaging applications, calendars, and/or clocks).

In some embodiments, the system 700 provides the functionality tocontrol or provide commands to the one or more computing devices 724based on a wearable device 702 determining motor actions or intendedmotor actions of the user. A motor action is an intended motor actionwhere before the user performs the motor action or before the usercompletes the motor action the detected neuromuscular signals travelingthrough the neuromuscular pathways can be determined to be the motoraction. Motor actions can be detected based on the detectedneuromuscular signals, but can additionally (using a fusion of thevarious sensor inputs), or alternatively, be detected using other typesof sensors (such as cameras focused on viewing hand movements and/orusing data from an inertial measurement unit that can detectcharacteristic vibration sequences or other data types to correspond toparticular in-air hand gestures). The one or more computing devices 724include one or more of a head-mounted display, smart phones, tablets,smart watches, laptops, computer systems, AR systems, robots, vehicles,virtual avatars, user interfaces, the wearable device 702, and/or otherelectronic devices and/or control interfaces.

The wearable device 702 includes a wearable structure worn by the user.In some embodiments, the wearable device 702 is an instance of thewrist-wearable device 104. In some embodiments, the wearable device 702collects information about a portion of the user's body (e.g., theuser's hand and finger position(s) and orientation(s)) that can be usedas input to perform one or more commands at the computing device 724. Insome embodiments, the collected information about a portion of theuser's body (e.g., the user's hand(s) and finger(s) position(s) andorientation(s)) can be used as input to perform one or more command atthe wearable device 724 (e.g., selecting content to present on theelectronic display 718 of the wearable device 702 or controlling one ormore applications 714 locally stored on the wearable device 702). Theinformation collected about the portion of the user's body includesneuromuscular signals that can be used by the one or more processors 704of the wearable device 702 to determine a motor action that the userintends to perform with their hands and/or fingers.

In the illustrated embodiment, the wearable device 702 includes the oneor more processors 704, memory 706, sensors 716, an electronic display718, a communication interface 720, and a learning module 722. In someembodiments, the memory 706 includes one or more of user profiles 708,motor actions 710, user-defined gestures 712, and applications 714. Thewearable device 702 may include additional components that are not shownin FIG. 7A, such as a power source (e.g., an integrated battery or aconnection to an external power source), a haptic feedback generator,etc. In some embodiments, one or more of the components shown in FIG. 7Aare housed within a conductive hemispherical shape of the wearabledevice.

In some embodiments, sensors 716 include one or more hardware devicesthat contact the user's skin (e.g., a wrist of a user 101). In someembodiments, the sensors 716 detect neuromuscular signals fromneuromuscular pathways within and on the wrist of a user (the sensorscan additionally, or alternatively, be coupled with the head-worndevices discussed herein). In some embodiments, the sensors 716 areconfigured to detect different digit movements, wrist movements, armmovements, index finger movements, middle finger movements, ring fingermovements, little finger movements, thumb movements, hand movements,etc. from the different neuromuscular signals detected from the user'sskin. In some embodiments, the sensors 716 are used in pairs to formrespective channels for detecting neuromuscular signals, where eachchannel is a pair of sensors. In some embodiments, the wearable device702 includes six pairs of sensors 716.

In some embodiments, the one or more processors 704 are configured toreceive the neuromuscular signals detected by the sensors 716 anddetermine a motor action 710. In some embodiments, each motor action 710is associated with one or more input commands. The input commands whenprovided to a computing device 724 cause the computing device to performan action (e.g., as described in detail above, various in-air handgestures can cause performance of various text-modification actions).Alternatively, in some embodiments the one or more input commands areused to cause the wearable device 702 to perform one or more actionslocally (e.g., present a display on the electronic display 718 and/oroperate one or more applications 714). For example, the wearable device702 can be a smart watch and the one or more input commands can be usedto cause the smart watch to perform one or more actions. In someembodiments, the motor action 710 and its associate input commands arestored in the memory 706. In some embodiments, the motor actions 710include digit movements, hand movements, wrist movements, arm movements,pinch gestures, index finger movements, middle finger movements, ringfinger movements, little finger movements, thumb movements, handclenches (or fists), waving motions, and/or other movements of theuser's hand or arm.

In some embodiments, the user can define one or more gestures using thelearning module 722. Specifically, in some embodiments the user canenter a training phase in which a user-defined gesture is associatedwith one or more input commands that when provided to a computing device724 cause the computing device to perform an action. Similarly, the oneor more input commands associated with the user-defined gesture can beused to cause the wearable device 702 to perform one or more actionslocally. The user-defined gesture, once trained, is stored in the memory706. Similar to the motor actions 710, the one or more processors 704can use the detected neuromuscular signals by the sensors 716 todetermine that a user-defined gesture was performed by the user.

The one or more applications 714 stored in the memory 706 can beproductivity-based applications (e.g., calendars, organizers, wordprocessors), social applications (e.g., social platforms), games, etc.In some embodiments, the one or more applications 714 are presented tothe user via the electronic display 718. In some embodiments, the one ormore applications 714 are used to facilitate the transmission ofinformation (e.g., to another application running on a computing device724). In some embodiments, the user can provide one or more inputcommands based on the determined motor action to the applications 714operating on the wearable device 702 to cause the applications 714 toperform the input commands.

Additionally, different user profiles 708 can be stored in the memory706. This allows the wearable device 702 to provide user-specificperformance. More specifically, the wearable device 702 can be tailoredto perform as efficiently as possible for each user.

The communication interface 720 enables input and output to thecomputing device 724. In some embodiments, the communication interface720 is a single communication channel, such as USB. In otherembodiments, the communication interface 720 includes several distinctcommunication channels operating together or independently. For example,the communication interface 720 may include separate communicationchannels for sending input commands to the computing device 724 to causethe computing device 724 to perform one or more actions. In someembodiments, data from the sensors 716 and/or the determined motoractions 710 are sent to the computing device 724, which then interpretsthe appropriate input response based on the received data. The one ormore communication channels of the communication interface 720 can beimplemented as wired or wireless connections. In some embodiments, thecommunication interface 720 includes hardware capable of datacommunications using any of a variety of custom or standard wirelessprotocols (e.g., IEEE 802.15.4, Wi-Fi, ZigBee, 6LoWPAN, Thread, Z-Wave,Bluetooth Smart, ISA100.11a, WirelessHART, or MiWi), custom or standardwired protocols (e.g., Ethernet or HomePlug), and/or any other suitablecommunication protocol, including communication protocols not yetdeveloped as of the filing date of this document.

In some embodiments, the computing device 724 presents media to a user.Examples of media presented by the computing device 724 include images,video, audio, or some combination thereof. Additional examples of mediainclude executed VR applications and/or AR applications to process inputdata from the sensors 716 on the wearable device 702. In someembodiments, the media content is based on received information from oneor more applications 732 (e.g., productivity applications, socialapplications, and/or games). The computing device 724 includes anelectronic display 726 for presenting media content to the user. Invarious embodiments, the electronic display 726 comprises a singleelectronic display or multiple electronic displays (e.g., one displayfor each eye of a user). The computing device 724 includes acommunication interface 728 that enables input and output to otherdevices in the system 700. The communication interface 728 is similar tothe communication interface 720 described above.

In some embodiments, the computing device 724 receives instructions (orcommands) from the wearable device 702. In response to receiving theinstructions, the computing device 724 performs one or more actionsassociated with the instructions (e.g., performs the one or more inputcommands in an AR or a VR environment). Alternatively, in someembodiments the computing device 724 receives instructions from anexternal device communicatively coupled to the wearable device 702 and,in response to receiving the instructions, performs one or more actionsassociated with the instructions. In some embodiments, the computingdevice 724 receives instructions from the wearable device 702 and, inresponse to receiving the instructions, provides the instruction to anexternal device communicatively coupled to the computing device 724,which performs one or more actions associated with the instructions.Although not shown, in the embodiments that include a distinct externaldevice, the external device may be connected to the wearable device 702and/or the computing device 724 via a wired or wireless connection. Theexternal device may be a remote game console, an additional display, anadditional head-mounted display, and/or any other additional electronicdevices that could be coupled in conjunction with the wearable device702 and/or the computing device 724.

In some embodiments, the computing device 724 provides information tothe wearable device 702, which in turn causes the wearable device topresent the information to the user. The information provided by thecomputing device 724 to the wearable device 702 can include mediacontent (which can be displayed on the electronic display 718 of thewearable device 702), organizational data (e.g., calendars, phonenumbers, invitation, directions), and files (e.g., word-processingdocuments, spreadsheets, or other documents that can be worked onlocally from the wearable device 702).

In some embodiments, the computing device 724 is implemented as anintegrated system-on-a-chip, a microcontroller, a desktop or laptopcomputer, a server computer, a tablet, or a smart phone or other mobiledevice. Thus, the computing device 724 includes components common totypical computing devices, such as the processor(s) 734, random-accessmemory (RAM), a storage device, a network interface, an input-out (I/O)interface, and the like. The processor may be or include one or moremicroprocessors or application-specific integrated circuits (ASICs). Thememory 730 may be or include RAM, ROM, DRAM, SRAM and MRAM and mayinclude firmware such as static data or fixed instructions, BIOS, systemfunctions, configuration data, and other routines used during theoperation of the computing device and the processor. The memory 730 alsoprovides a storage area for data and instructions associated withapplications and data handled by the processor.

The storage device provides non-volatile, bulk, or long-term storage ofdata or instructions in the computing device. The storage device maytake the form of a magnetic or solid-state disk, tape, CD, DVD, or otherreasonably high-capacity addressable or serial storage medium. Multiplestorage devices may be provided or available to the computing device.Some of these storage devices may be external to the computing device,such as network storage or cloud-based storage. The network interfaceincludes an interface to a network and can be implemented as either awired or wireless interface. The I/O interface interfaces the processorto peripherals (not shown) such as, for example and depending upon thecomputing device, sensors, displays, cameras, color sensors,microphones, keyboards, and USB devices.

In the example shown in FIG. 7A, the computing device 724 furtherincludes applications 732. In some embodiments, the applications 732 areimplemented as software modules that are stored on the storage deviceand executed by the processor 734. Some embodiments of the computingdevice 724 include additional or different components than thosedescribed in conjunction with FIG. 7A. Similarly, the functions furtherdescribed below may be distributed among components of the computingdevice 724 in a different manner than is described here.

Each application 732 is a group of instructions that, when executed by aprocessor, generates specific content for presentation to the user. Forexample, an application 732 can include a VR application that generatesVR content (such as a VR environment) and that further generates VRcontent in response to inputs received from the wearable device 702(based on determined user motor actions). Examples of VR applicationsinclude gaming applications, conferencing applications, and videoplayback applications. Additional examples of applications 732 caninclude productivity-based applications (e.g., calendars, organizers andword processors), social-based applications (e.g., social mediaplatforms and dating platforms), entertainment (e.g., shows, games, andmovies), and travel (e.g., ride-share applications, hotel applications,and airline applications).

In some embodiments, the computing device 724 allows the applications732 to operate in conjunction with the wearable device 702. In someembodiments, the computing device 724 receives information from thesensors 716 of the wearable device 702 and provides the information toan application 732. Based on the received information, the application732 determines media content to provide to the computing device 724 (orthe wearable device 702) for presentation to the user via the electronicdisplay 726 and/or a type of haptic feedback. For example, if thecomputing device 724 receives information from the sensors 716 on thewearable device 702 indicating that the user has performed an action(e.g., performed a sword slash in a game, opened a file, or typed amessage), the application 732 generates content for the computing device724 (or the wearable device 702) to present, with the content mirroringthe user's instructions based on determined motor actions by thewearable device 702. Similarly, in some embodiments the applications 732receive information directly from the sensors 716 on the wearable device702 (e.g., applications locally saved to the wearable device 702) andprovide media content to the computing device 724 for presentation tothe user based on the information (e.g., determined motor actions by thewearable device 702).

FIG. 7B is a block diagram illustrating a system 750 in accordance withsome embodiments. While some example features are illustrated, variousother features have not been illustrated for the sake of brevity and soas not to obscure pertinent aspects of the example embodiments disclosedherein. To that end, as a non-limiting example, the system 750 includesaccessory devices 752-1 and 752-2 (e.g., wearable devices), which areused in conjunction with a computer system 772 (e.g., a computing device724).

An example accessory device 752 includes, for example, one or moreprocessors/cores 754 (referred to henceforth as processors), a memory756, one or more actuators 760, one or more communications components764, and/or one or more sensors 758. In some embodiments, thesecomponents are interconnected by way of a communications bus 766.References to these components of the accessory device 752 coverembodiments in which one or more of these components (and combinationsthereof) are included. In some embodiments, the one or more sensors 758and the one or more transducers 762 are the same components. In someembodiments, the example accessory device 752 includes one or morecameras 770. In some embodiments (not shown), the accessory device 752includes a wearable structure. In some embodiments, the accessory deviceand the wearable structure are integrally formed. In some embodiments,the accessory device and the wearable structure are distinct structuresyet are part of the system 750. In some embodiments, one or more of theaccessory devices 752 is the wrist-wearable device 104.

For example, the accessory device 752-1 may be a ring that is used inconjunction with a wearable structure to utilize data measurementsobtained by sensor 758-1 to adjust a fit of the wearable structure. Inanother example, the accessory device 752-1 and accessory device 752-2are distinct wristbands to be worn on each wrist of the user.

In some embodiments, a single processor 754 (e.g., processor 754-1 ofthe accessory device 752-1) executes software modules for controllingmultiple accessory devices 752 (e.g., accessory devices 752-1 . . .752-n). In some embodiments, a single accessory device 752 (e.g.,accessory device 752-2) includes multiple processors 754 (e.g.,processors 754-2) such as one or more actuator processors, one or morecommunications component processors, one or more sensor processors,and/or one or more transducer processors. In some embodiments, the oneor more actuator processors are configured to adjust a fit of a wearablestructure. In some embodiments, the one or more communicationsprocessors are configured to control communications transmitted bycommunications component 764 and/or receive communications by way ofcommunications component 764. In some embodiments, the one or moresensor processors are configured to control operation of sensor 758and/or receive output from sensors 758. In some embodiments, the one ormore transducer processors are configured to control the operation oftransducers 762.

In some embodiments, the communications component 764 of the accessorydevice 752 includes a communications component antenna for communicatingwith the computer system 772. In some embodiments, the communicationscomponent 774 includes a complementary communications component antennathat communicates with the communications component 764. In someembodiments, the data contained within the communication signals alertsthe computer system 772 that the accessory device 752 is ready for use.In some embodiments, the computer system 772 sends instructions to theaccessory device 752 and, in response to receiving the instructions, theaccessory device 752 instructs a transmit electrode and receiveelectrode to provide coupling information between the receive electrodeand the user.

In some embodiments, the one or more actuators 760 are used to adjust afit of the wearable structure on a user's appendage. In someembodiments, the one or more actuators 760 are also used to providehaptic feedback to the user. For example, each actuator 760 may applyvibration stimulations, pressure stimulations, shear stimulations, orsome combination thereof to the user. In some embodiments, the one ormore actuators 760 are hydraulic, pneumatic, electric, and/or mechanicalactuators.

In some embodiments, the one or more transducers 762 are used totransmit and receive one or more signals 768. In some embodiments, theone or more sensors 758 are used to transmit and receive one or moresignals 768. In some embodiments, the one or more sensors 758 and theone or more transducers 762 are part of a same component that is used totransmit and receive one or more signals 768. The signals 768 may beelectromagnetic waves, mechanical waves, electrical signals, or anywave/signal capable of being transmitted through a medium. As usedherein, a medium includes the wearer's skin, flesh, bone, blood vessels,or some combination thereof.

In addition to transmitting signals (e.g., electrical signals), theaccessory device 752 is also configured to receive (e.g., detect, sense)signals transmitted by itself or by another accessory device 752. Toillustrate, a first accessory device 752-1 may transmit a plurality ofsignals through a medium, such as a user's appendage, and a secondaccessory device 752-2 may receive the signals transmitted by the firstaccessory device 752-1 through the medium. Furthermore, an accessorydevice 752 receiving transmitted signals may use the received signals todetermine whether the accessory device is in contact with a user.

In some embodiments, the one or more transducers 762 of the accessorydevice 752-1 include one or more transducers configured to generateand/or receive signals. In some embodiments, integrated circuits (notshown) of the accessory device 752-1, such as a controller circuitand/or signal generator, control the behavior of the transducers 762. Insome embodiments, the transmit electrode and/or the receive electrodeare part of the one or more transducers 762 of the accessory device752-1. Alternatively, the transmit electrode and/or the receiveelectrode may be part of the one or more sensors 758-1 of the accessorydevice 752-1, or the transmit electrode may be part of a transducer 762while the receive electrode may be part of a sensor 758-1 (or viceversa).

In some embodiments, the sensors 758 include one or more of the transmitelectrode and the receive electrode for obtaining coupling information.In some embodiments, the sensors 758 includes one or more neuromuscularsensors. In some embodiments, the neuromuscular sensors include one ormore surface electromyography (sEMG) sensors, mechanomyography (MMG)sensors, and/or sonomyography (SMG) sensors. Additional non-limitingexamples of the sensors 758 (and the sensors 790) include infrared,pyroelectric, ultrasonic, microphone, laser, optical, Doppler, gyro,accelerometer, resonant LC sensors, capacitive sensors, acousticsensors, and/or inductive sensors. In some embodiments, the sensors 758(and the sensors 790) are configured to gather additional data about theuser (e.g., an impedance of the user's body). Examples of sensor dataoutput by these sensors include body temperature data, infraredrange-finder data, motion data, activity recognition data, silhouettedetection and recognition data, gesture data, heart rate data, and otherwearable-device data (e.g., biometric readings and output, accelerometerdata).

The computer system 772 is a computing device that executesartificial-reality applications (e.g., VR applications and/or ARapplications) to process input data from the sensors 790 on thehead-mounted display 782 and the sensors 758 on the accessory device752. The computer system 772 provides output data to at least (i) theelectronic display 784 on the head-mounted display 782 and (ii) theaccessory device(s) 752. In some embodiments, the head-mounted display782 is an instance of the head-mounted display 102 or the AR glasses202.

The computer system 772 includes one or more processors/cores 776, thememory 778, one or more communications components 774, and/or one ormore cameras 780. In some embodiments, these components areinterconnected by way of a communications bus 794. References to thesecomponents of the computer system 772 cover embodiments in which one ormore of these components (and combinations thereof) are included.

In some embodiments, the computer system 772 is a stand-alone devicethat is coupled to a head-mounted display 782. For example, the computersystem 772 has processor(s)/core(s) 776 for controlling one or morefunctions of the computer system 772 and the head-mounted display 782has processor(s)/core(s) 786 for controlling one or more functions ofthe head-mounted display 782. Alternatively, in some embodiments thehead-mounted display 782 is a component of the computer system 772. Forexample, the processor(s) 776 control functions of the computer system772 and the head-mounted display 782. In addition, in some embodimentsthe head-mounted display 782 includes the processor(s) 786 thatcommunicate with the processor(s) 776 of the computer system 772. Insome embodiments, communications between the computer system 772 and thehead-mounted display 782 occur via a wired (or wireless) connectionbetween communications bus 794 and communications bus 792. In someembodiments, the computer system 772 and the head-mounted display 782share a single communications bus. In some embodiments, the head-mounteddisplay 782 is separate from the computer system 772.

The computer system 772 may be any suitable computer device, such as alaptop computer, a tablet device, a netbook, a personal digitalassistant, a mobile phone, a smart phone, an artificial-reality consoleor device (e.g., a VR device, an AR device, or the like), a gamingdevice, a computer server, or any other computing device. The computersystem 772 is sometimes called a host or a host system. In someembodiments, the computer system 772 includes other user interfacecomponents such as a keyboard, a touch-screen display, a mouse, atrack-pad, and/or any number of supplemental I/O devices to addfunctionality to computer system 772.

In some embodiments, one or more cameras 780 of the computer system 772are used to facilitate the artificial-reality experience. In someembodiments, the computer system 772 provides images captured by the oneor more cameras 780 to the display 784 of the head-mounted display 782,and the display 784 in turn displays the provided images. In someembodiments, the processors 786 of the head-mounted display 782 processthe provided images. It is noted that in some embodiments one or more ofthe cameras 780 are part of the head-mounted display 782.

The head-mounted display 782 presents media to a user. Examples of mediapresented by the head-mounted display 782 include images, video, audio,or some combination thereof. In some embodiments, audio is presented viaan external device (e.g., speakers and/or headphones) that receivesaudio information from the head-mounted display 782, the computer system772, or both, and presents audio data based on the audio information.The displayed images may be in VR, AR, or mixed reality. The display 784displays images to the user in accordance with data received from thecomputer system 772. In various embodiments, the display 784 comprises asingle electronic display or multiple electronic displays (e.g., onedisplay for each eye of a user).

The sensors 790 include one or more hardware devices that detect spatialand motion information about the head-mounted display 782. In someembodiments, the sensors 790 include one or more neuromuscular sensors.In some embodiments, the neuromuscular sensors include one or more sEMGsensors, MMG sensors, and/or SMG sensors. Spatial and motion informationcan include information about the position, orientation, velocity,rotation, and acceleration of the head-mounted display 782. For example,the sensors 790 may include one or more inertial measurement units thatdetect rotation of the user's head while the user is wearing thehead-mounted display 782. In some embodiments, the sensors 790 includeone or more cameras positioned on the head-mounted display 782. In someembodiments, the head-mounted display 782 includes one or more sensors790. In some embodiments, one or more of the sensors 790 are part of thecomputer system 772.

Having thus described example block diagrams, attention will now bedirected to examples of the wearable devices that can be used inconjunction with the techniques described herein.

Example Wearable Devices (e.g., Wrist-Wearable Devices, Such as SmartWatches, AR Glasses, and VR Goggles/Headsets, which can Form VariousArtificial-Reality Systems Used with the Techniques Described Herein)

FIG. 8A shows an example augmented-reality (AR) system 820 in accordancewith some embodiments. In FIG. 8A, the AR system 820 includes an eyeweardevice with a frame 824 configured to hold a left display device 828-1and a right display device 828-2 in front of a user's eyes. The displaydevices 828-1 and 828-2 may act together or independently to present animage or series of images to a user. While the AR system 820 includestwo displays, embodiments of this disclosure may be implemented in ARsystems with a single near-eye display (NED) or more than two NEDs. Insome embodiments, the AR system 820 is an instance of the AR glasses202.

In some embodiments, the AR system 820 includes one or more sensors,such as the sensors 830 and 832 (e.g., instances of the sensors 716 ofFIG. 7A). For example, the sensors 830 and 832 may generate measurementsignals in response to motion of the AR system 820 and may be located onsubstantially any portion of the frame 810. Each sensor may be aposition sensor, an inertial measurement unit (IN/U), a depth cameraassembly, or any combination thereof. In some embodiments, the AR system820 includes more or fewer sensors than is shown in FIG. 8A. Inembodiments in which the sensors include an IMU, the IMU may generatecalibration data based on measurement signals from the sensors. Examplesof the sensors include, without limitation, accelerometers, gyroscopes,magnetometers, other suitable types of sensors that detect motion,sensors used for error correction of the IMU, or some combinationthereof.

In some embodiments, the AR system 820 includes a microphone array witha plurality of acoustic sensors 826-1 through 826-8, referred tocollectively as the acoustic sensors 826. The acoustic sensors 826 maybe transducers that detect air pressure variations induced by soundwaves. In some embodiments, each acoustic sensor 826 is configured todetect sound and convert the detected sound into an electronic format(e.g., an analog or digital format). In some embodiments, the microphonearray includes 10 acoustic sensors: 826-1 and 826-2, designed to beplaced inside a corresponding ear of the user; acoustic sensors 826-3,826-4, 826-5, 826-6, 826-7, and 826-8 positioned at various locations onthe frame 824; and acoustic sensors positioned on a correspondingneckband, where the neckband is an optional component of the system thatis not present in certain embodiments of the artificial-reality systemsdiscussed herein. In some embodiments, the neckband is an example of acomputing device 724 or the computer system 772.

The configuration of the acoustic sensors 826 of the microphone arraymay vary. While the AR system 820 discussed with reference to FIG. 8Ahas 10 acoustic sensors 826, the number of acoustic sensors 826 may begreater or less than 10. In some situations, using more acoustic sensors826 increases the amount of audio information collected and/or thesensitivity and accuracy of the audio information. In contrast, in somesituations, using a lower number of acoustic sensors 826 decreases thecomputing power required by a controller 836 to process the collectedaudio information. In addition, the position of each acoustic sensor 826of the microphone array may vary. For example, the position of anacoustic sensor 826 may include a defined position on the user, adefined coordinate on the frame 824, an orientation associated with eachacoustic sensor, or some combination thereof.

The acoustic sensors 826-1 and 826-2 may be positioned on differentparts of the user's ear, such as behind the pinna or within the auricleor fossa. In some embodiments, there are additional acoustic sensors onor surrounding the ear in addition to acoustic sensors 826 inside theear canal. In some situations, having an acoustic sensor positioned nextto an ear canal of a user enables the microphone array to collectinformation on how sounds arrive at the ear canal. By positioning atleast two of the acoustic sensors 826 on either side of a user's head(e.g., as binaural microphones), the AR device 820 is able to simulatebinaural hearing and capture a three-dimensional (3D) stereo sound fieldaround a user's head. In some embodiments, the acoustic sensors 826-1and 826-2 are connected to the AR system 820 via a wired connection and,in other embodiments, the acoustic sensors 826-1 and 826-2 are connectedto the AR system 820 via a wireless connection (e.g., a Bluetoothconnection). In some embodiments, the AR system 820 does not include theacoustic sensors 826-1 and 826-2.

The acoustic sensors 826 on the frame 824 may be positioned along thelength of the temples, across the bridge, above or below the displaydevices 828, or in some combination thereof. The acoustic sensors 826may be oriented such that the microphone array is able to detect soundsin a wide range of directions surrounding the user wearing AR system820. In some embodiments, a calibration process is performed during themanufacture of the AR system 820 to determine relative positioning ofeach acoustic sensor 826 in the microphone array.

In some embodiments, the eyewear device further includes, or iscommunicatively coupled to, an external device (e.g., a paired device)such as the optional neckband discussed above. In some embodiments, theoptional neckband is coupled to the eyewear device via one or moreconnectors. The connectors may be wired or wireless connectors and mayinclude electrical and/or non-electrical (e.g., structural) components.In some embodiments, the eyewear device and the neckband operateindependently without any wired or wireless connection between them. Insome embodiments, the components of the eyewear device and the neckbandare located on one or more additional peripheral devices paired with theeyewear device, the neckband, or some combination thereof. Furthermore,the neckband is intended to represent any suitable type or form ofpaired device. Thus, the following discussion of the neckband may alsoapply to various other paired devices such as smart watches, smartphones, wristbands, other wearable devices, handheld controllers, tabletcomputers, or laptop computers.

In some situations, pairing external devices, such as the optionalneckband, with the AR eyewear device enables the AR eyewear device toachieve the form factor of a pair of glasses while still providingsufficient battery and computation power for expanded capabilities. Someor all of the battery power, computational resources, and/or additionalfeatures of the AR system 820 may be provided by a paired device orshared between a paired device and an eyewear device, thus reducing theweight, heat profile, and form factor of the eyewear device overallwhile still retaining desired functionality. For example, the neckbandmay allow components that would otherwise be included on an eyeweardevice to be included in the neckband, thereby shifting a weight loadfrom a user's head to a user's shoulders. In some embodiments, theneckband has a larger surface area over which to diffuse and disperseheat to the ambient environment. Thus, the neckband may allow forgreater battery and computation capacity than might otherwise have beenpossible for a stand-alone eyewear device. Because weight carried in theneckband may be less invasive to a user than weight carried in theeyewear device, a user may tolerate wearing a lighter eyewear device andcarrying or wearing the paired device for greater lengths of time thanthe user would tolerate wearing a heavy stand-alone eyewear device,thereby enabling an artificial-reality environment to be incorporatedmore fully into a user's day-to-day activities.

In some embodiments, the optional neckband is communicatively coupledwith the eyewear device and/or to other devices (e.g., thewrist-wearable device 104). The other devices may provide certainfunctions (e.g., tracking, localizing, depth mapping, processing, and/orstorage) to the AR system 820. In some embodiments, the neckbandincludes a controller and a power source. In some embodiments, theacoustic sensors of the neckband are configured to detect sound andconvert the detected sound into an electronic format (analog ordigital).

The controller of the neckband processes information generated by thesensors on the neckband and/or the AR system 820. For example, thecontroller may process information from the acoustic sensors 826. Foreach detected sound, the controller may perform a direction-of-arrivalestimation to estimate a direction from which the detected sound arrivedat the microphone array. As the microphone array detects sounds, thecontroller may populate an audio data set with the information. Inembodiments in which the AR system 820 includes an IMU, the controller836 may compute all inertial and spatial calculations from the IMUlocated on the eyewear device. The connector may convey informationbetween the eyewear device and the neckband and between the eyeweardevice and the controller. The information may be in the form of opticaldata, electrical data, wireless data, or any other transmittable dataform. Moving the processing of information generated by the eyeweardevice to the neckband may reduce weight and heat in the eyewear device,making it more comfortable and safer for a user to use.

In some embodiments, the power source in the neckband provides power tothe eyewear device and the neckband. The power source may include,without limitation, lithium-ion batteries, lithium-polymer batteries,primary lithium batteries, alkaline batteries, or any other form ofpower storage. In some embodiments, the power source is a wired powersource.

As noted, some artificial-reality systems may, instead of blending anartificial reality with actual reality, substantially replace one ormore of a user's sensory perceptions of the real world with a virtualexperience. One example of this type of system is a head-worn displaysystem, such as the virtual-reality (VR) system 850 in FIG. 8B, whichmostly or completely covers a user's field of view.

FIG. 8B shows a VR system 850 (e.g., also referred to herein as VRgoggles or a VR headset) in accordance with some embodiments. The VRsystem 850 includes the head-mounted display 102. The head-mounteddisplay 102 includes a front body 852 and a frame 854 (e.g., a strap orband) shaped to fit around a user's head. In some embodiments, thehead-mounted display 102 includes output audio transducers 856-1 and856-2, as shown in FIG. 8B. In some embodiments, the front body 852and/or the frame 854 includes one or more electronic elements, includingone or more electronic displays, one or more IMUs, one or more trackingemitters or detectors, and/or any other suitable device or sensor forcreating an artificial-reality experience.

Artificial-reality systems may include a variety of types of visualfeedback mechanisms. For example, display devices in the AR system 820and/or the VR system 850 may include one or more liquid-crystaldisplays, light-emitting diode (LED) displays, organic LED displays,and/or any other suitable type of display screen. Artificial-realitysystems may include a single display screen for both eyes or may providea display screen for each eye, which may allow for additionalflexibility for varifocal adjustments or for correcting a user'srefractive error. Some artificial-reality systems also include opticalsubsystems having one or more lenses (e.g., conventional concave orconvex lenses, Fresnel lenses, and/or adjustable liquid lenses) throughwhich a user may view a display screen.

In addition to or instead of using display screens, someartificial-reality systems include one or more projection systems. Forexample, display devices in the AR system 820 and/or the VR system 850may include micro-LED projectors that project light (e.g., using awaveguide) into display devices, such as clear combiner lenses thatallow ambient light to pass through. The display devices may refract theprojected light toward a user's pupil and may enable a user tosimultaneously view both artificial-reality content and the real world.Artificial-reality systems may also be configured with any othersuitable type or form of image-projection system.

Artificial-reality systems may also include various types of computervision components and subsystems. For example, the systems 820 and 850may include one or more optical sensors such as two-dimensional (2D) or3D cameras, time-of-flight depth sensors, single-beam or sweeping laserrangefinders, 3D LiDAR sensors, and/or any other suitable type or formof optical sensor. An artificial-reality system may process data fromone or more of these sensors to identify a location of a user, to mapthe real world, to provide a user with context about real-worldsurroundings, and/or to perform a variety of other functions.

Artificial-reality systems may also include one or more input and/oroutput audio transducers. In the examples shown in FIG. 8C, the outputaudio transducers 856 may include voice coil speakers, ribbon speakers,electrostatic speakers, piezoelectric speakers, bone conductiontransducers, cartilage conduction transducers, and/or any other suitabletype or form of audio transducer. Similarly, the input audio transducersmay include condenser microphones, dynamic microphones, ribbonmicrophones, and/or any other type or form of input transducer. In someembodiments, a single transducer is used for both audio input and audiooutput.

In some embodiments, the artificial-reality systems 820 and 850 includehaptic (tactile) feedback systems, which may be incorporated intoheadwear, gloves, body suits, handheld controllers, environmentaldevices (e.g., chairs or floor mats), and/or any other type of device orsystem, such as the wearable devices discussed herein. The hapticfeedback systems may provide various types of cutaneous feedback,including vibration, force, traction, shear, texture, and/ortemperature. The haptic feedback systems may also provide various typesof kinesthetic feedback, such as motion and compliance. The hapticfeedback may be implemented using motors, piezoelectric actuators,fluidic systems, and/or a variety of other types of feedback mechanisms.The haptic feedback systems may be implemented independently of otherartificial-reality devices, within other artificial-reality devices,and/or in conjunction with other artificial-reality devices.

FIGS. 9A-9C illustrate examples of wearable devices in accordance withsome embodiments. FIG. 9A illustrates a wearable device 900 inaccordance with some embodiments. The wrist-wearable device 104 shownand described in reference to FIGS. 1A-4E can be an instance of thewearable device 900. FIG. 9A illustrates a perspective view of thewearable device 900 that includes a device body 902 decoupled from adevice band 904. The device body 902 and the device band 904 areconfigured to allow a user to wear the wearable device 900 on a bodypart (e.g., a wrist). The wearable device 900 includes a retainingmechanism 963 (e.g., a buckle or a hook and loop fastener) for securingthe device band 904 to the user's body. The wearable device 900 alsoincludes a coupling mechanism 914 (e.g., a cradle) for detachablycoupling the device body 902 (via a coupling surface 912 of the devicebody 902) to the device band 904.

Functions executed by the wearable device 900 can include, withoutlimitation, display of visual content to the user (e.g., visual contentdisplayed on display screen 901), sensing user input (e.g., sensing atouch on button 916, sensing biometric data on sensor 918, or sensingneuromuscular signals on neuromuscular sensor 920), messaging (e.g.,text, speech, video), image capture, wireless communications (e.g.,cellular, near field, Wi-Fi, personal area network), locationdetermination, financial transactions, providing haptic feedback,alarms, notifications, biometric authentication, health monitoring,sleep monitoring, etc. These functions can be executed independently inthe device body 902, independently in the device band 904, and/or incommunication between the device body 902 and the device band 904. Insome embodiments, functions can be executed on the wearable device 900in conjunction with an artificial-reality environment.

In some embodiments, the device band 904 is configured to be worn by auser such that an inner surface of the device band 904 is in contactwith the user's skin. Thus, when worn by a user, the sensor 918 is incontact with the user's skin. In some embodiments, the sensor 918 is abiosensor that senses a user's heart rate, saturated oxygen level,temperature, sweat level, muscle intentions, or a combination thereof.In some embodiments, the device band 904 includes multiple sensors 918that can be distributed on an inside and/or an outside surface of thedevice band 904. Additionally, or alternatively, the device body 902includes the same or different sensors than the device band 904. Thedevice body 902 (e.g., a capsule portion) can include, withoutlimitation, a magnetic field sensor, antenna return loss sensor,front-facing image sensor 908 and/or a rear-facing image sensor, abiometric sensor, an IMU, a heart rate sensor, a saturated oxygensensor, a neuromuscular sensor(s), an altimeter sensor, a temperaturesensor, a bioimpedance sensor, a pedometer sensor, an optical sensor, atouch sensor, and/or a sweat sensor, among others. The sensor 918 canalso include a sensor that provides data about a user's environment suchas a user's motion (e.g., an IMU), altitude, location, orientation,gait, or a combination thereof. The sensor 918 can also include a lightsensor (e.g., an infrared light sensor, a visible light sensor) that isconfigured to track a position and/or motion of the device body 902and/or the device band 904. In some embodiments, the device band 904transmits the data acquired by the sensor 918 to the device body 902using a wired communication method (e.g., a UART, a USB transceiver)and/or a wireless communication method (e.g., near-field communication,Bluetooth™). In some embodiments, the device band 904 is configured tooperate (e.g., to collect data using sensor 918) independent of whetherthe device body 902 is coupled to or decoupled from the device band 904.

The device band 904 includes a haptic device 922 (e.g., a vibratoryhaptic actuator) that is configured to provide haptic feedback (e.g., acutaneous and/or kinesthetic sensation) to the user's skin. The sensor918 and/or the haptic device 922 can be configured to operate inconjunction with multiple applications including, without limitation,health monitoring, social media, game playing, and artificial reality(e.g., the applications associated with artificial reality).

In some embodiments, the device band 904 includes a neuromuscular sensor920 (e.g., an EMG sensor, an MMG sensor, an SMG sensor). Theneuromuscular sensor 920 senses a user's intention to perform certainmotor actions. The sensed muscle intention can be used to controlcertain user interfaces displayed on the display 901 and/or can betransmitted to a device responsible for rendering an artificial-realityenvironment (e.g., the head-mounted display device 102) to perform anaction in an associated artificial-reality environment, such as tocontrol the motion of a virtual device displayed to the user.

In some embodiments, signals from the neuromuscular sensor 920 are usedto provide a user with an enhanced interaction with a physical objectand/or a virtual object in an artificial-reality application generatedby an artificial-reality system. Although FIG. 9A shows oneneuromuscular sensor 920, the device band 904 can include a plurality ofneuromuscular sensors 920 arranged circumferentially on an insidesurface of the device band 904 such that the plurality of neuromuscularsensors 920 contact the skin of the user. The neuromuscular sensor 920senses and records neuromuscular signals from the user as they performmuscular activations (e.g., movements, gestures). The muscularactivations performed by the user can include static gestures, such asplacing the user's hand palm down on a table; dynamic gestures, such asgrasping a physical or virtual object; and covert gestures that areimperceptible to another person, such as slightly tensing a joint byco-contracting opposing muscles or using sub-muscular activations. Themuscular activations performed by the user can include symbolic gestures(e.g., gestures mapped to other gestures, interactions, or commands, forexample, based on a gesture vocabulary that specifies the mapping ofgestures to commands).

As shown in the example of FIG. 9A, the device band coupling mechanism914 can include a type of frame or shell that allows the couplingsurface 912 to be retained within the device band coupling mechanism914. The device body 902 can be detachably coupled to the device band904 through a friction fit, magnetic coupling, a rotation-basedconnector, a shear-pin coupler, a retention spring, one or more magnets,a clip, a pin shaft, a hook and loop fastener, or any combinationthereof. In some embodiments, the device body 902 is decoupled from thedevice band 904 by actuation of a release mechanism 910. The releasemechanism 910 can include, without limitation, a button, a knob, aplunger, a handle, a lever, a fastener, a clasp, a dial, a latch, or anycombination thereof.

FIG. 9B illustrates a wearable device 970, in accordance with someembodiments. In some embodiments, the wrist-wearable device 104 is aninstance of the wearable device 970. In some embodiments, the wearabledevice 970 is used to generate control information (e.g., sensed dataabout neuromuscular signals or instructions to perform certain commandsafter the data is sensed) for causing a computing device (e.g., thecomputer system 772) to perform one or more input commands. In someembodiments, the wearable device 970 includes a plurality ofneuromuscular sensors 976. In some embodiments, the plurality ofneuromuscular sensors 976 includes a predetermined number (e.g., 16) ofneuromuscular sensors (e.g., EMG sensors) arranged circumferentiallyaround an elastic band 974. The plurality of neuromuscular sensors 976may include any suitable number of neuromuscular sensors. In someembodiments, the number and arrangement of neuromuscular sensors 976depend on the particular application for which the wearable device 970is used. For instance, a wearable device 970 configured as an armband,wristband, or chest-band may include a plurality of neuromuscularsensors 976 with a different number of neuromuscular sensors anddifferent arrangement for each use case, such as medical use cases ascompared to gaming or general day-to-day use cases. For example, atleast 16 neuromuscular sensors 976 may be arranged circumferentiallyaround the elastic band 974.

In some embodiments, the elastic band 974 is configured to be wornaround a user's lower arm or wrist. The elastic band 974 may include aflexible electronic connector 972. In some embodiments, the flexibleelectronic connector 972 interconnects separate sensors and electroniccircuitry that are enclosed in one or more sensor housings.Alternatively, in some embodiments, the flexible electronic connector972 interconnects separate sensors and electronic circuitry that areoutside of the one or more sensor housings. Each neuromuscular sensor ofthe plurality of neuromuscular sensors 976 can include a skin-contactingsurface that includes one or more electrodes. One or more sensors of theplurality of neuromuscular sensors 976 can be coupled using flexibleelectronics incorporated into the wearable device 970.

FIG. 9C illustrates a wearable device 979 in accordance with someembodiments. In some embodiments, the wrist-wearable device 104 is aninstance of the wearable device 979. The wearable device 979 includespaired sensor channels 985 a-985 f along an interior surface of awearable structure 975 that are configured to detect neuromuscularsignals. Different number of paired sensors channels can be used (e.g.,one pair of sensors, three pairs of sensors, four pairs of sensors, sixpairs of sensors). The wearable structure 975 can include a band portion990, a capsule portion 995, and a cradle portion (not pictured) that iscoupled with the band portion 990 to allow for the capsule portion 995to be removably coupled with the band portion 990. For embodiments inwhich the capsule portion 995 is removable, the capsule portion 995 canbe referred to as a removable structure, such that in these embodimentsthe wearable device includes a wearable portion (e.g., the band portion990 and the cradle portion) and a removable structure (the removablecapsule portion which can be removed from the cradle). In someembodiments, the capsule portion 995 includes the one or more processorsand/or other components of the wearable device 702 described above inreference to FIG. 7A. The wearable structure 975 is configured to beworn by a user 101. More specifically, the wearable structure 975 isconfigured to couple the wearable device 979 to a wrist, an arm, aforearm, or other portion of the user's body. Each paired sensorchannels 985 a-985 f includes two electrodes 980 (e.g., electrodes 980a-980 h) for sensing neuromuscular signals based on differential sensingwithin each respective sensor channel. In accordance with someembodiments, the wearable device 970 further includes an electricalground and a shielding electrode.

The techniques described above can be used with any device for sensingneuromuscular signals, including the arm-wearable devices of FIG. 9A-9C,but could also be used with other types of wearable devices for sensingneuromuscular signals (such as body-wearable or head-wearable devicesthat might have neuromuscular sensors closer to the brain or spinalcolumn).

Having thus described system-block diagrams and then example wearabledevices, attention will now be directed to certain example embodiments.

Example Embodiments

Turning now to some example embodiments of the methods, devices,systems, and computer-readable storage media described earlier. Inshort, the descriptions below proceed by first discussing the paragraphsbeginning with an A symbol, which are related to an aspect in whichin-air hand gestures are used to allow for initiating and interactingwith a text-modification mode; following that is a discussion ofparagraphs beginning with a B symbol, which relate to an aspect in whicha specific in-air hand gesture is used to activate a microphone toenable inputting of voice-provided text.

(A1) In one aspect, some embodiments include a method (e.g., the method500) of modifying text. The method is performed at a wearable device(e.g., the wearable device 702) having a memory (e.g., memory 706) andone or more processors (e.g., processor(s) 704). The method includes (i)causing display, using a display that is in communication with awearable device (e.g., the display 718 or the display 726), of aplurality of text terms input by a user (e.g., the draft message 110,FIG. 1A); (ii) detecting, using data from one or moreneuromuscular-signal sensors (e.g., the sensors 716) in communicationwith the wearable device, an in-air hand gesture performed by the userwhile the plurality of text terms are displayed (e.g., the gesture 120,FIG. 1B); (iii) in response to the in-air hand gesture, enabling atext-modification mode that allows for modifying the plurality of textterms input by the user; and (iv) while the text-modification mode isenabled (a) identifying a target term of the plurality of text terms(e.g., the term 130, FIG. 1C), (b) receiving data about a voice inputprovided by the user for modifying the target term (e.g., the spokenreplacement term 134, FIG. 1D), and (c) causing a modification to thetarget term in accordance with the voice input from the user (e.g., thereplacement term 136, FIG. 1D). In some embodiments, the target term isidentified before the text-modification mode is enabled. In someembodiments, the target term is identified prior to detecting the in-airhand gesture.

In some embodiments, the wearable device is a wrist-wearable device,such as a smart watch (e.g., the wrist-wearable device 104). In someembodiments, the wearable device includes an inertial measuring unit(IMU). In some embodiments, the wearable device is a head-worn wearabledevice, such as smart glasses (e.g., the augmented-reality (AR) glasses202). The methods described herein, in addition to being performed at awearable device, can also be performed at an artificial-reality system(e.g., the system 700) that includes both a wrist-wearable device and ahead-worn wearable device, among other hardware accessories orcomponents.

In some embodiments, by using the data from the one or moreneuromuscular-signal sensors, an in-air hand gesture described herein isdetected before its performance has been completed by the user, such asdetecting an intention to perform the in-air hand gesture followed bycompletion of the in-air hand gesture, which can either be separatelydetected or determined based on the user's intention to perform thein-air hand gesture.

In some embodiments, the in-air hand gestures described herein includemovement of a user's wrist, elbow, and shoulder (e.g., an arm lift orwave gesture). In some embodiments, the in-air gesture is atext-modification-initiation gesture (e.g., a double pinch gesture). Insome embodiments, the in-air gesture is a gesture that does not contactthe wearable device.

(A2) The method of A1, where the target term is identified based on agaze of the user (e.g., using an AR/virtual-reality (VR) gaze-trackingcomponent). For example, a user's gaze is tracked using the camera(s)770 and/or the camera(s) 780 of the system 750.

(A3) The method of A1 or A2, where (i) the target term is identified inaccordance with detection, using data from the one or moreneuromuscular-signal sensors, of an additional in-air hand gestureperformed by the user; and (ii) the additional in-air hand gesture isdistinct from the in-air hand gesture (e.g., a tap, swipe, or scrollgesture). In some embodiments, the additional in-air hand gesture is athumb d-pad-like movement, detected via the neuromuscular-signalsensors, to indicate which word should be modified. In some embodiments,the target term is highlighted using a first gesture (e.g., ad-pad-style thumb movement), then selected using a second gesture (e.g.,an in-air force pinch gesture). In some embodiments, an IMU is used tocontrol a cursor to select the target term.

(A4) The method of any of A1-A3, further including, while thetext-modification mode is enabled, (i) tracking a user gaze and (ii)causing the target term to be emphasized on the display in accordancewith the user gaze, where identifying the target term includesdetecting, using data from the one or more neuromuscular-signal sensors,an additional in-air hand gesture while the target term is emphasized.In some embodiments, emphasizing the target term includes one or more ofhighlighting, bolding, changing text style, changing text color,underlining, italicizing, and the like (e.g., the first in-air gestureis a held pinch, and the second in-air gesture is a harder pinch). Insome embodiments, the plurality of text terms are displayed in a displaymode and wherein the user gaze is not tracked while in the display mode.In some embodiments, gaze tracking is performed using one or more of acontact lens with embedded sensors for measuring eye movements,electrode(s) placed in proximity to the user's eyes (e.g., usingelectrooculography), and optical sensors (e.g., for detectingreflections in a user's eyes, such as corneal reflections). In someembodiments, the optical sensors track eye features to monitor eyemovement and determine gaze direction.

(A5) The method of A4, where the user gaze is not tracked while thetext-modification mode is disabled. In some embodiments, the user gazeis not tracked while in a text-display mode. For example, the gazetracking components are disabled (powered off) to conserve power on thewearable device and increase user privacy.

(A6) The method of any of A1-A5, further including, while thetext-modification mode is enabled, causing a first term of the pluralityof text terms to be emphasized on the display, where (i) the first termappears before the target term in the plurality of text terms, (ii) thevoice input is received while the first term is emphasized on thedisplay, and (iii) the voice input is determined to modify a phrase thatincludes the first term and the target term. For example, the phrase“pick up Sally” is modified to “pick up Calli” where “pick” is the firstterm and “Sally” is the target term. In some embodiments, the targetterm is identified from context and term type from a subset of termsnear the first term (e.g., in a same paragraph or line, or within 20,10, or 5 words of the first term). In some embodiments, the phrasebegins with the first term.

(A7) The method of any of A1-A6, where the target term is identifiedbased on a determination that a term type of the target term matches aclassification of the voice input from the user. For example, the termtype of the target term is matched to the classification of the voiceinput using a word classifier and context for the voice input. In someembodiments, the target term is identified using automatic speechrecognition to determine that the voice input from the user is a wordthat is of a same type as the target word (e.g., the voice inputcomprises a proper noun, a day, a date, a time, a number, or alocation). In some embodiments, the target term is identified as ahomophone of the voice input from the user.

(A8) The method of any of A1-A7, further including, prior to causingdisplay of the plurality of text terms, (i) detecting, using data fromthe one or more neuromuscular-signal sensors, an additional in-air handgesture performed by the user, the additional in-air hand gesture beingdistinct from the in-air hand gesture; (ii) in response to theadditional in-air hand gesture, enabling an input mode; and (iii) whilein the input mode, receiving data about the plurality of text termsinput by the user and causing the display of each of the plurality oftext terms as the data is received. For example, FIGS. 4B-4D illustratethe user 101 providing inputs while holding the gesture 408. FIGS. 4B-4Dfurther show text 410 and 422 being displayed in the messengerapplication 108 in response to the provided inputs.

(A9) The method of A8, where the plurality of text terms are receivedvia additional voice inputs provided by the user. For example, in FIG.4C the user 101 provides voice inputs 409 that are converted to the text410.

(A10) The method of any of A1-A9, where the plurality of text termsinput by the user are caused to be displayed on a display of thewearable device. For example, FIG. 2E shows a draft message 206displayed to the user 101 with the terms in the phrase 232 correspondingto the replacement phrase 230.

(A1 l) The method of any of A1-A10, where the wearable device is awrist-wearable device that is configured to send instructions to ahead-worn wearable device that includes the display. For example, thewearable device is the wrist-wearable device 104, 900, 970, or 979. Asan example, the head-worn wearable device is the head-mounted displaydevice 102 or the AR glasses 202.

(A12) The method of any of A1-A11, further including: (i) aftermodifying the target term, exiting the text-modification mode; and (ii)after exiting the text-modification mode and in response to detecting anadditional in-air hand gesture using data from the one or moreneuromuscular-signal sensors, causing the plurality of text terms,including the target term after it has been modified, to be sent to oneor more other users via a messaging application. For example, theadditional gesture is a double thumb swipe-right gesture. In someembodiments, exiting includes transitioning to a different mode such asa text-display mode in which text is viewable but is not editable untilthe text-modification mode is re-enabled. In some embodiments, sendingthe plurality of text terms is in accordance with a two-step processincluding a first gesture to request the sending and a second gesture toconfirm the sending. This two-step process can help to avoidcircumstances in which a user might accidentally send a message beforethey are ready to do so. In addition to, or as an alternative to thetwo-step process, the additional in-air hand gesture that is used tocause the sending of a message can be selected to ensure that accidentalsending incidents occur less frequently, such as by selecting an in-airhand gesture that is less likely to be accidentally performed by theuser and/or detected by the system.

(A13) The method of any of A1-A12, further including, while thetext-modification mode is enabled: (i) detecting, using data from theone or more neuromuscular-signal sensors, an additional in-air handgesture performed by the user, the additional in-air hand gesture beingdistinct from the in-air hand gesture; and (ii) in response to theadditional in-air hand gesture, causing the display of a context menu inproximity to the plurality of text terms. For example, FIG. 3B shows theuser 101 performing the gesture 320 and the context menu 322 beingdisplayed in response. In some embodiments, the context menu includesoptions to copy, cut, and/or paste text.

(A14) The method of A13, further including, while the text-modificationmode is enabled: (i) detecting, using data from the one or moreneuromuscular-signal sensors, one more in-air hand gesture performed bythe user, the one more in-air hand gesture being distinct from thein-air hand gesture and the additional in-air hand gesture; and (ii) inresponse to the one more in-air hand gesture, causing display of one ormore modifiers. For example, FIG. 3F shows the user 101 performing thegesture 356 and the modifiers menu 357 being displayed in response.

(A15) The method of any of A1-A14, further including, while thetext-modification mode is enabled: (i) detecting, using data from theone or more neuromuscular-signal sensors, an additional in-air handgesture performed by the user; and (ii) in response to the additionalin-air hand gesture, causing a deletion of one or more terms of theplurality of text terms from being displayed on the display. Forexample, FIG. 3E shows the user 101 performing the gesture 350 (e.g., afist gesture) and the emphasized term 342 in FIG. 3D having been deletedin FIG. 3E in response to the gesture 350. In some embodiments,performance of the deletion gesture (e.g., the fist gesture) when acaret is placed will cause the character at the caret in the term to bedeleted. In some embodiments, performance of the deletion gesture when aterm is selected will cause the entire term to be deleted.

(A16) The method of any of A1-A15, further including, while thetext-modification mode is enabled: (i) detecting, using data from theone or more neuromuscular-signal sensors, an additional in-air handgesture performed by the user, the additional in-air hand gesture beinga same or different gesture from the in-air hand gesture; and (ii) inresponse to the additional in-air hand gesture, exiting thetext-modification mode. For example, FIG. 3H shows the user 101performing the gesture 360 and the word-processing application 301disabling the text-modification mode in response to the gesture 360. Insome embodiments, the additional in-air gesture is an exit gesture,e.g., a gesture that performs an action that is similar to actionperformed after a press of an escape (ESC) key or back command in anymode.

(A17) The method of any of A1-A16, further including, while causing thedisplay of the plurality of text terms, causing the display of arepresentation of one or more available gesture commands. For example,FIG. 3A shows the actions menu 308 with each action 310 in the actionsmenu 308 including an indication of the corresponding gesture for theuser 101.

(A18) The method of A17, where the one or more available gesturecommands are based on an operational mode of the wearable device and therepresentation is updated as the operational mode of the wearable devicechanges. In some embodiments, the one or more available gestures includeall available gesture commands. In some embodiments, all availablegesture commands are ranked, and the one or more available gesturecommands include one or more top-ranked commands (e.g., ranked bylikelihood of use). For example, the plurality of actions 364 in FIG. 3His different from the plurality of actions in the actions menu 308 inFIG. 3A due to the word-processing application 301 being in a differentmode (e.g., text-modification mode being enabled in FIG. 3A and disabledin FIG. 3H).

(A19) The method of any of A1-A18, further including, while thetext-modification mode is enabled: (i) causing a first term of theplurality of text terms to be emphasized on the display; (ii) whilecausing the first term to be emphasized, detecting, using data from theone or more neuromuscular-signal sensors, an additional in-air handgesture performed by the user; and (iii) in response to the additionalin-air hand gesture, displaying a menu with one or more replacementoptions for the first term, wherein the one or more replacement optionsare obtained from a speech model. For example, FIG. 3B shows the user101 performing the gesture 320 and shows the context menu 322 includingthe plurality of options 324 in response to the gesture 320. In someembodiments, the one or more replacement options include one or moreterms and one or more graphical objects (e.g., emojis).

(B1) In another aspect, some embodiments include a method (e.g., themethod 600) of inputting text. The method is performed at a wearabledevice (e.g., the wearable device 702) having memory (e.g., memory 706)and one or more processors (e.g., processor(s) 704). The methodincludes: (i) while data associated with a messaging application (e.g.,the messenger application 108) is displayed on a display (e.g., thedisplay 718 or the display 726) that is in communication with thewearable device, detecting, using data from one or moreneuromuscular-signal sensors (e.g., the sensors 716) in communicationwith the wearable device, an in-air hand gesture performed by the user(e.g., the gesture 408, FIG. 4B); (ii) in response to the in-air handgesture, enabling a microphone to capture audio for use in conjunctionwith the messaging application; (iii) receiving data about a voice inputfrom the user via the microphone (e.g., the voice inputs 409); (iv)converting the voice input to a plurality of text terms (e.g., the text410); and (v) causing a display, within the messaging application on thedisplay, of the plurality of text terms to the user. For example, FIG.4B shows the user 101 performing the gesture 408 while the messengerapplication 108 is displayed and FIG. 4C shows the user 101 providingthe voice inputs 409 and the text 410 corresponding to the voice inputs409 in the new message 402.

(B2) The method of B1, where the in-air hand gesture is a gesture duringwhich the user's thumb is held against a user's digit for at least apredetermined period (e.g., a thumb and middle finger pinch gesture),and the method further includes disabling the microphone in response todetecting release of the in-air hand gesture. For example, the gesture408 in FIGS. 4B-4D is a thumb and index finger pinch gesture. Tocontinue the example, FIG. 4E shows the gesture 408 having been releasedand the microphone being disabled in response (as denoted by the statusmessage 434).

(B3) The method of B1, where the in-air hand gesture is a toggle gesturethat is detected at a first point in time, and the method furtherincludes disabling the microphone in response to a subsequent detectionof the in-air hand gesture at a second point in time that is after thefirst point in time. In some embodiments, the in-air hand gesture is adouble thumb and middle finger pinch gesture (e.g., to enable themicrophone). In some embodiments, a second in-air hand gesture issubsequently detected (e.g., a flick gesture) to disable the microphone(e.g., after providing voice inputs).

(B4) The method of any of B1-B3, further including: (i) while causingthe display of the plurality of text terms within the messagingapplication, detecting, using data from the one or moreneuromuscular-signal sensors, an additional in-air hand gesture; (ii) inresponse to the additional in-air gesture, enabling a text-modificationmode; (iii) while the text-modification mode is enabled, tracking a gazeof the user with respect to the plurality of text terms displayed withinthe messaging application; and (iv) causing a term of the plurality oftext terms to be emphasized in accordance with the tracking of the gazeof the user. For example, FIG. 2B shows the augmented-reality glasses202 tracking the gaze of the user 101 (e.g., as denoted by the gazelines 214). FIG. 2B further shows the term 215 emphasized in accordancewith the user gaze tracking. The wearable device identifies the targetterm by detecting, using data from the one or more neuromuscular-signalsensors, a fourth in-air hand gesture while the target term isemphasized. For example, FIG. 2D shows the emphasized term 216 from FIG.2C selected in accordance with the gesture 220 and the gaze-trackinglines 214. In some embodiments, the emphasized term is selected inaccordance with a selection gesture (e.g., a thumb and index fingerpinch gesture). In some embodiments, the selected term is moved (e.g.,dragged) by the user holding the selection gesture and moving their handand inserted at a new location by the user releasing the selectiongesture. In some embodiments, the emphasized term is indicated by theuser with placement of a caret (e.g., using a gaze-tracking and a caretplacement gesture, such as a thumb and index finger pinch gesture). Insome embodiments, the selection gesture is a double thumb and indexfinger pinch gesture.

(B5) The method of B4, further including, while the text-modificationmode is enabled, (i) detecting, using data from the one or moreneuromuscular-signal sensors, one more in-air hand gesture performed bythe user, and, (ii) in response to the one more in-air hand gesture,causing a deletion of the emphasized term. For example, FIG. 3E showsthe user 101 performing the gesture 350 and the emphasized term 342 inFIG. 3D having been deleted in FIG. 3E in response to the gesture 350.

(B6) The method of B4 or B5, where the gaze of the user is not trackedprior to enabling the text-modification mode. In some embodiments, usergaze tracking is disabled while in one or more non-text-modificationmodes (e.g., a text-input mode or text-display mode). In someembodiments, user gaze tracking is only enabled while thetext-modification mode is enabled.

(B7) The method of any of B1-B6, further including: (i) in response tothe in-air hand gesture, enabling an input mode; (ii) while the inputmode is enabled and after converting the voice input to the plurality oftext terms, detecting, using data from the one or moreneuromuscular-signal sensors, an additional in-air hand gestureperformed by the user; and (iii) in response to the additional in-airhand gesture, causing a deletion of a term of the plurality of textterms. For example, FIG. 4F shows the user 101 performing the gesture440 and the corresponding deletion of the last term in the message 433(“stop”) in FIG. 4E.

(B8) The method of any of B1-B7, further including: (i) after causing adisplay of the plurality of text terms to the user, detecting, usingdata from the one or more neuromuscular-signal sensors, an additionalin-air hand gesture, the additional in-air hand gesture being distinctfrom the in-air hand gesture; and (ii) in response to detecting theadditional in-air hand gesture, causing the plurality of text terms tobe sent to one or more other users via the messaging application. Forexample, FIG. 1F shows the user performing the gesture 146 and the draftmessage 110 from FIG. 1E being sent to the person “M” in response to thegesture 146.

(B9) The method of any of B1-B8, where the wearable device is awrist-wearable device that is configured to send instructions to ahead-worn wearable device that includes the display. For example, thewearable device is the wrist-wearable device 104, 900, 970, or 979. Asan example, the head-worn wearable device is the head-mounted displaydevice 102 or the augmented-reality glasses 202.

(B10) The method of any of B1-B8, where the wearable device is ahead-mounted device that is configured to communicate with one or moreadditional wearable devices. For example, the wearable device is thehead-mounted display device 102 or the augmented-reality glasses 202.

In some embodiments, one or more of the gestures described above (e.g.,with respect to A1-A19 and B1-B10) are detected with an optical sensor(e.g., a camera) or sensors associated with an inertial measurement unit(IMU) rather than (or in addition to, via fusing the sensor inputs todetect the various in-air hand gestures described herein) the one ormore neuromuscular-signal sensors. In some embodiments, the one or moregestures described above (e.g., with respect to A1-A19 and B1-B10) arereplaced with gestures performed by other parts of the user's body(e.g., head gestures, leg gestures, torso gestures). As one example, thein-air hand gesture used to activate a text-modification mode can be anod of the user's head, which can be detected using one or moreneuromuscular-signal sensors, data from an IMU, and cameras; as anotherexample, the deletion gesture described above can be a shaking of theuser's head (as if the user is indicating “No”). In some embodiments,the wearable device detects neuromuscular signals traveling through theuser's neck or back, which can be done using neuromuscular-signalsensors coupled with the VR goggles or the AR glasses in some exampleembodiments. In some embodiments, the one or more gestures describedabove (e.g., with respect to A1-A19 and B1-B10) are replaced with (orperformed using) in-air hand gestures on a controller (e.g., a handheldcontroller or foot pedal controller). In some embodiments, the one ormore in-air hand gestures described above (e.g., with respect to A1-A19and B1-B10) are replaced with audio commands (e.g., spoken word commandsor non-word commands such as a tongue click).

In another aspect, some embodiments include a computing system includingone or more processors and a memory coupled to the one or moreprocessors, the memory storing one or more programs configured to beexecuted by the one or more processors, and the one or more programsincluding instructions for performing any of the methods describedherein (e.g., methods 500, 600, A1-A19, or B1-B10 above).

In yet another aspect, some embodiments include a non-transitorycomputer-readable storage medium storing one or more programs forexecution by one or more processors of a computing system, the one ormore programs including instructions for performing any of the methodsdescribed herein (e.g., methods 500, 600, A1-A19, or B1-B10 above).

While text terms are used as a primary illustrative example herein, theskilled artisan will appreciate upon reading this disclosure that theinventive techniques discussed herein can also be used to allow forneuromuscular gesture control of additional types of inputs, includinggraphical inputs (such as images including near any of the text terms oremojis), attachment-type inputs (e.g., document attachments), and manyother types of inputs that can be provided at wearable devices inaddition to text-based inputs.

It will be understood that although the terms “first,” “second,” etc.may be used herein to describe various elements, these elements shouldnot be limited by these terms. These terms are only used to distinguishone element from another.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the claims. Asused in the description of the embodiments and the appended claims, thesingular forms “a,” “an” and “the” are intended to include the pluralforms as well, unless the context clearly indicates otherwise. It willalso be understood that the term “and/or” as used herein refers to andencompasses any and all possible combinations of one or more of theassociated listed items. It will be further understood that the terms“comprises” and/or “comprising,” when used in this specification,specify the presence of stated features, integers, steps, operations,elements, and/or components, but do not preclude the presence oraddition of one or more other features, integers, steps, operations,elements, components, and/or groups thereof.

As used herein, the term “if” can be construed to mean “when,” “upon,”or “in response to determining,” “in accordance with a determination” or“in response to detecting” that a stated condition precedent is true,depending on the context. Similarly, the phrase “if it is determined[that a stated condition precedent is true]” or “if [a stated conditionprecedent is true]” or “when [a stated condition precedent is true]” canbe construed to mean “upon determining,” “in response to determining,”“in accordance with a determination,” “upon detecting” or “in responseto detecting” that the stated condition precedent is true, depending onthe context.

The foregoing description, for the purpose of explanation, has beendescribed with reference to specific embodiments. However, theillustrative discussions above are not intended to be exhaustive orlimit the claims to the precise forms disclosed. Many modifications andvariations are possible in view of the above teachings. The embodimentswere chosen and described in order to best explain principles ofoperation and practical applications to thereby enable others skilled inthe art.

What is claimed is:
 1. A method performed at a wearable device havingmemory and one or more processors, the method comprising: causingdisplay, using a display that is in communication with a wearabledevice, of a plurality of text terms input by a user; detecting, usingdata from one or more neuromuscular-signal sensors in communication withthe wearable device, an in-air hand gesture performed by the user whilethe plurality of text terms are displayed; in response to the in-airhand gesture, enabling a text-modification mode that allows formodifying the plurality of text terms input by the user; identifying atarget term of the plurality of text terms; and while thetext-modification mode is enabled: receiving data about a voice inputprovided by the user for modifying the target term; and causing amodification to the target term in accordance with the voice input fromthe user.
 2. The method of claim 1, wherein the target term isidentified based on a gaze of the user.
 3. The method of claim 1,wherein: the target term is identified in accordance with detection,using data from the one or more neuromuscular-signal sensors, of anadditional in-air hand gesture performed by the user, and the additionalin-air hand gesture is distinct from the in-air hand gesture.
 4. Themethod of claim 1, further comprising, while the text-modification modeis enabled: tracking a user gaze; and causing the target term to beemphasized on the display in accordance with the user gaze, whereinidentifying the target term comprises detecting, using data from the oneor more neuromuscular-signal sensors, an additional in-air hand gesturewhile the target term is emphasized.
 5. The method of claim 4, whereinthe user gaze is not tracked while the text-modification mode isdisabled.
 6. The method of claim 1, further comprising, while thetext-modification mode is enabled, causing a first term of the pluralityof text terms to be emphasized on the display, wherein: the first termappears before the target term in the plurality of text terms, the voiceinput is received while the first term is emphasized on the display, andthe voice input is determined to modify a phrase that includes the firstterm and the target term.
 7. The method of claim 1, wherein the targetterm is identified based on a determination that a term type of thetarget term matches a classification of the voice input from the user.8. The method of claim 1, further comprising, prior to causing displayof the plurality of text terms: detecting, using data from the one ormore neuromuscular-signal sensors, an additional in-air hand gestureperformed by the user, the additional in-air hand gesture being distinctfrom the in-air hand gesture; in response to the additional in-air handgesture, enabling an input mode; and while in the input mode, receivingdata about the plurality of text terms input by the user and causing thedisplay of each of the plurality of text terms as the data is received.9. The method of claim 1, wherein the plurality of text terms input bythe user are caused to be displayed on a display of the wearable device.10. The method of claim 1, wherein the wearable device is awrist-wearable device that is configured to send instructions to ahead-worn wearable device that includes the display.
 11. The method ofclaim 1, further comprising: after modifying the target term, exitingthe text-modification mode; and after exiting the text-modification modeand in response to detecting an additional in-air hand gesture usingdata from the one or more neuromuscular-signal sensors, causing theplurality of text terms, including the target term after it has beenmodified, to be sent to one or more other users via a messagingapplication.
 12. The method of claim 1, further comprising, while thetext-modification mode is enabled: detecting, using data from the one ormore neuromuscular-signal sensors, an additional in-air hand gestureperformed by the user, the additional in-air hand gesture being distinctfrom the in-air hand gesture; and in response to the additional in-airhand gesture, causing display of a context menu in proximity to theplurality of text terms.
 13. The method of claim 12, further comprising,while the text-modification mode is enabled: detecting, using data fromthe one or more neuromuscular-signal sensors, one more in-air handgesture performed by the user, the one more in-air hand gesture beingdistinct from the in-air hand gesture and the additional in-air handgesture; and in response to the one more in-air hand gesture, causingdisplay of one or more modifiers.
 14. The method of claim 1, furthercomprising, while the text-modification mode is enabled: detecting,using data from the one or more neuromuscular-signal sensors, anadditional in-air hand gesture performed by the user; and in response tothe additional in-air hand gesture, causing a deletion of one or moreterms of the plurality of text terms from being displayed on thedisplay.
 15. The method of claim 1, further comprising, while thetext-modification mode is enabled: detecting, using data from the one ormore neuromuscular-signal sensors, an additional in-air hand gestureperformed by the user, the additional in-air hand gesture being a sameor different gesture from the in-air hand gesture; and in response tothe additional in-air hand gesture, exiting the text-modification mode.16. The method of claim 1, further comprising, while causing display ofthe plurality of text terms, causing display of a representation of oneor more available gesture commands.
 17. The method of claim 16, whereinthe one or more available gesture commands are based on an operationalmode of the wearable device and the representation is updated as theoperational mode of the wearable device changes.
 18. The method of claim1, further comprising, while the text-modification mode is enabled:causing a first term of the plurality of text terms to be emphasized onthe display; while causing the first term to be emphasized, detecting,using data from the one or more neuromuscular-signal sensors, anadditional in-air hand gesture performed by the user; and in response tothe additional in-air hand gesture, displaying a menu with one or morereplacement options for the first term, wherein the one or morereplacement options are obtained from a speech model.
 19. Anartificial-reality system, comprising: one or more processors; memorycoupled to the one or more processors, the memory storing instructionsfor: causing display, using a display that is in communication with awearable device, of a plurality of text terms input by a user;detecting, using data from one or more neuromuscular-signal sensors incommunication with the wearable device, an in-air hand gesture performedby the user while the plurality of text terms are displayed; in responseto the in-air hand gesture, enabling a text-modification mode thatallows for modifying the plurality of text terms input by the user;identifying a target term of the plurality of text terms; and while thetext-modification mode is enabled: receiving data about a voice inputprovided by the user for modifying the target term; and causing amodification to the target term in accordance with the voice input fromthe user.
 20. A non-transitory computer-readable storage medium storingone or more programs configured for execution by a computing devicehaving one or more processors and memory, the one or more programscomprising instructions for: causing display, using a display that is incommunication with a wearable device, of a plurality of text terms inputby a user; detecting, using data from one or more neuromuscular-signalsensors in communication with the wearable device, an in-air handgesture performed by the user while the plurality of text terms aredisplayed; in response to the in-air hand gesture, enabling atext-modification mode that allows for modifying the plurality of textterms input by the user; identifying a target term of the plurality oftext terms; and while the text-modification mode is enabled: receivingdata about a voice input provided by the user for modifying the targetterm; and causing a modification to the target term in accordance withthe voice input from the user.